www.bpbonline.com

www.bpbonline.com

作者简介

About the Author

Indrajit Kar是一位杰出的AI思想领袖，著有5本AI/ML书籍，是一位创新者和作家，拥有超过22年的经验，致力于推动各行业变革性的AI产品和平台。在他的职业生涯中，他领导过众多高影响力团队，负责开发AI、ML、GenAI和数据科学领域的端到端解决方案，指导项目从概念化和设计到部署和扩展的整个过程。

Indrajit Kar is a distinguished AI thought leader, author of 5 AI/ML books, innovator, and author with over 22 years of experience driving transformative AI-led products and platforms across industries. Throughout his career, he has led numerous high-impact teams responsible for developing end-to-end solutions in AI, ML, GenAI, and data science - guiding projects from conceptualization and design to deployment and scaling.

作为现任人工智能主管，Indrajit 领导着多个大型项目，为众多全球客户带来可衡量的业务影响。他的工作根植于其在全人类人工智能 (GenAI)、大型语言模型( LLM ) 架构、MLOps、自然语言处理和计算机视觉等领域的深厚技术专长。他在将 LLM 和自主人工智能代理集成到电子商务、医疗保健、生命科学、电信和制造业等多个行业的实际应用中发挥了关键作用。

In his current role as head of AI, Indrajit spearheads large-scale initiatives that deliver measurable business impact across a diverse portfolio of global clients. His work is rooted in deep technical expertise across GenAI, large language model (LLM) architectures, MLOps, natural language processing, and computer vision. He has played a key role in integrating LLMs and autonomous AI agents into real-world applications spanning sectors such as e-commerce, healthcare, life sciences, telecommunications, and manufacturing.

Indrajit 同时也是企业高管的战略顾问和合作伙伴，致力于帮助企业通过先进的人工智能产品和平台转型释放商业价值。他始终引领潮流，弥合前沿研究与企业级实施之间的鸿沟，加速人工智能在各组织中的应用。

Indrajit is also a strategic advisor and collaborator to C-level executives, helping enterprises unlock business value through advanced AI product and platform transformations. His leadership consistently bridges the gap between cutting-edge research and enterprise-scale implementation, accelerating AI adoption across organizations.

作为人工智能领域公认的权威人物，Indrajit著有两本书，其中一本专门探讨了基因人工智能及其行业应用。他在人工智能研究领域也贡献卓著，发表了27篇以上的论文，申请了21项专利，并荣获多项殊荣，包括来自知名会议和机构的八项最佳论文奖。他的研究常常探索创新、可扩展性和负责任的人工智能之间的交集。

A recognized voice in the AI community, Indrajit has authored two books, including one dedicated to GenAI and its industry applications. He has also contributed extensively to AI research, with 27+ published papers, 21 patents filed, and multiple accolades, including eight Best Paper Awards from reputed conferences and institutions. His work often explores the intersections of innovation, scalability, and responsible AI.

凭借在领导研发项目以及为财富 500 强企业管理人工智能服务和产品化工作的丰富经验，Indrajit 将继续引领智能系统的未来发展。他对创新的热情，以及对符合伦理且可扩展的人工智能的愿景，驱动着他通过变革性技术赋能企业和社区的使命。

With a legacy of leading R&D programs and having managed AI services and productization efforts for Fortune 500 companies, Indrajit continues to shape the future of intelligent systems. His passion for innovation, combined with a vision for ethical and scalable AI, drives his mission to empower businesses and communities through transformative technology.

关于评论者

About the Reviewers

• Dhanveer Singh 是美国第一资本银行的一位技术领导者，在金融服务、保险和零售行业拥有超过 19 年的软件工程、云架构和大规模系统现代化经验。他擅长 AWS、微服务、容器化、DevOps、大数据和人工智能/机器学习，致力于打造安全、高性能的平台，处理数十亿笔交易，服务全球数百万用户。

  作为云原生架构和自动化领域的倡导者，Dhanveer 领导了云成本优化、弹性工程和网络安全自动化方面的变革性举措，显著提升了效率并推动了企业数字化转型。他还拥有多项数据集成、转换、数据安全和云自动化领域的专利，彰显了他对创新的重视。

  除了在技术领导方面做出贡献外，他还担任国际期刊和会议的审稿人和技术程序委员会成员，担任全球 IT 和网络安全奖项的评委，并通过 STEM 和 CodeDay 项目提供指导。

  Dhanveer 是 IETE 和 IAENG 的会士，也是 IEEE 和 ACM 的活跃成员。

• Dhanveer Singh is a technology leader at Capital One USA with over 19 years of experience in software engineering, cloud architecture, and large-scale system modernization across financial services, insurance, and retail. He specializes in AWS, microservices, containerization, DevOps, big data, and AI/ML, delivering secure, high-performing platforms that process billions of transactions and serve millions worldwide.

  An advocate of cloud-native architectures and automation, Dhanveer has led transformative initiatives in cloud cost optimization, resilience engineering, and cybersecurity automation, driving measurable efficiency and advancing enterprise digital transformation. He has also filed multiple patents in areas of data integration, transformation, data security, and cloud automation, underscoring his focus on innovation.

  Beyond his technical leadership, he contributes as a reviewer and TPC member for international journals and conferences, serves as a judge for global IT and cybersecurity awards, and mentors through STEM and CodeDay programs.

  Dhanveer is a Fellow of IETE and IAENG, and an active IEEE and ACM member.

• 哈文德拉·辛格是一位杰出的技术领袖，专长于云工程、架构、自动化和人工智能解决方案。他运用 Azure、.NET、C#、Python、GCP、Kubernetes、Databricks 和其他前沿技术，设计并实现可扩展、安全的系统。凭借在云原生应用、微服务、事件驱动架构和分布式系统方面的专业知识，哈文德拉致力于推动云和人工智能生态系统的创新，提供能够创造商业价值并实现可持续增长的高影响力解决方案。

• Harvendra Singh is a distinguished technology leader specializing in cloud engineering, architecture, automation, and AI-powered solutions. He designs and implements scalable, secure systems utilizing Azure, .NET, C#, Python, GCP, Kubernetes, Databricks, and other cutting-edge technologies. With expertise in cloud-native applications, microservices, event-driven architectures, and distributed systems, Harvendra drives innovation in cloud and AI ecosystems, delivering high-impact solutions that drive business value and sustainable growth.

• 马尼什·贾恩 (Manish Jain)是 Firstsource Solutions 的副总裁兼人工智能架构负责人，负责领导财富 100 强企业的企业级人工智能转型。他拥有超过 20 年的技术领导经验，其中包括十余年推动先进人工智能创新，并因此被公认为能够带来可量化业务影响的变革性解决方案架构师。除了公司职责外，他还担任 Deeplearning.ai 的技术顾问，并在 Analytics Vidhya 担任导师。此外，他还为 Manning 等知名人工智能出版社审稿。以及 Packt 出版社，使他成为研究与企业实际应用交叉领域的佼佼者。Manish 兼具深厚的技术专长和卓越的领导才能，能够指导企业完成人工智能转型中的战略和运营层面的工作。他对推动人工智能社区发展的承诺体现在他的咨询和指导工作中，以及他参与同行评审期刊的发表。

  这些经验使他成为人工智能转型的必要性以及在复杂的企业环境中扩展人工智能的实际挑战方面的权威，始终将创新与可衡量的结果联系起来。

• Manish Jain is the vice president and head of AI architecture at Firstsource Solutions, where he leads enterprise-wide AI transformation for Fortune 100 organizations. With more than 20 years of technology leadership, including over a decade driving advanced AI innovation, he has earned recognition as an architect of transformative solutions that deliver quantifiable business impact. In addition to corporate responsibilities, his acts as a technical consultant for Deeplearning.ai and mentors at Analytics Vidhya. He also serves as a manuscript reviewer for prominent AI publishers such as Manning and Packt, positioning him at the crossroads of research and practical enterprise applications. Manish is unique blend of deep technical expertise and proven executive leadership enables him to guide organizations through the strategic and operational aspects of AI transformation. His commitment to advancing the AI community is evident in his advisory and mentoring roles, as well as his involvement in peer-reviewed publishing.

  These experiences make a compelling authority on the imperatives of AI transformation and the practical challenges of scaling AI across complex enterprise environments, consistently linking innovation with measurable outcomes.

致谢

Acknowledgement

我衷心感谢我的家人、父母、妻子、岳父母和孩子们，他们坚定不移的鼓励和信任是我这段旅程的基石。衷心感谢BPB出版社的耐心和信任，使得本书得以分册出版，全面深入地涵盖了人工智能领域瞬息万变的方方面面。我也感谢我的公司，感谢他们为我提供发展机会，让我得以开发基因人工智能和智能体应用，这些都为本书分享的见解提供了宝贵的素材。对于所有支持我的人，无论你们是否在场，你们的指导和鼓励都深深地影响了我的这段旅程，对此我将永远铭记于心。

I extend my deepest appreciation to my family, parents, wife, in-laws, and children, whose steadfast encouragement and belief in me have been the cornerstone of this journey. Heartfelt thanks to BPB Publications for their patience and trust, allowing the book’s multi-part publication to thoroughly cover the dynamic field of AI. I am also grateful to my companies for fostering growth and providing opportunities to develop GenAI and agentic applications, which informed the insights shared here. To everyone who supported me, seen and unseen, your guidance and encouragement have profoundly shaped this journey, for which I am eternally thankful.

前言

Preface

我们生活在智能协作时代，人工智能不再仅仅是工具，而是能够检索知识、生成想法、推理问题并跨文本、图像和语音等多种模态进行交互的伙伴。多模态和智能体应用的出现标志着我们构建、部署和依赖人工智能的方式发生了转折。

We are living in the age of intelligent collaboration, where AI is no longer just a tool, but a partner capable of retrieving knowledge, generating ideas, reasoning through problems, and interacting across modalities like text, images, and voice. The emergence of multimodal and agentic applications marks a turning point in how we build, deploy, and rely on AI.

本书《构建多模态生成式人工智能和智能体应用》是一本实用指南，旨在帮助读者超越理论，真正构建未来人工智能系统。全书共18章，循序渐进地从基础知识入手，逐步深入到高级实现，首先介绍检索、生成和编排；然后讲解结合文本、图像和语音的多模态工作流程；最后探讨文本到SQL系统、OCR、欺诈检测和人工智能运维等实际应用。

This book, Building Multimodal Generative AI and Agentic Applications, is a practical guide for those who want to move beyond theory and actually build the future of AI systems. Across 18 chapters, you will move step-by-step from fundamentals to advanced implementations, starting with retrieval, generation, and orchestration; progressing into multimodal workflows that combine text, images, and voice; and then advancing toward real-world applications like text-to-SQL systems, OCR, fraud detection, and AI operations.

每一章都注重实践性和易懂性。您将找到概念解释、系统设计原则、代码示例以及练习题，这些练习将引导您进行实验并在实践中学习。

Every chapter is designed to be hands-on and approachable. You will find conceptual explanations, system design principles, code walkthroughs, and to do exercises that push you to experiment and learn by doing.

本书的目标不仅是解释这些系统是如何工作的，而且还要赋予你构建自己的可扩展、多模态和智能AI应用程序的能力，这些应用程序可靠、安全且具有影响力。

The goal of this book is not only to explain how these systems work, but also to empower you to build your own scalable, multimodal, and agentic AI applications, applications that are reliable, safe, and impactful.

无论您是工程师、研究人员还是技术领导者，我都希望这本书能为您提供塑造下一代人工智能所需的知识、信心和灵感。

Whether you are an engineer, researcher, or leader in technology, I hope that this book equips you with the knowledge, confidence, and inspiration to shape the next-generation of AI.

第一章：新时代生成式人工智能简介 ——本章介绍现代人工智能系统的关键组成部分。首先概述生成式人工智能，然后探讨检索系统、生成系统及其各自的优势。本章阐述了检索增强生成（RAG ）如何将两者结合起来，以及编排如何帮助不同的人工智能组件协同工作。此外，本章还解释了标记、向量数据库和重排序方法，以及双向编码器和交叉编码器之间的区别。最后，本章讨论了诸如安全使用人工智能的防护措施、代理的作用以及模型上下文协议的重要性等重要主题。

Chapter 1: Introducing New Age Generative AI - This chapter introduces the key building blocks of modern AI systems. It begins with an overview of generative AI and then explores retrieval systems, generation systems, and the strengths of each. It covers how retrieval-augumented generation (RAG) generation combines the two, and how orchestration helps different AI components work together. The chapter also explains tokens, vector databases, and reranking methods, along with the differences between bi-encoders and cross-encoders. Finally, it discusses essential topics like guardrails for safe AI use, the role of agents, and the importance of Model Context Protocols.

第二章：深入探讨多模态系统——本章重点介绍视觉语言模型及其在多模态人工智能中的作用。它解释了什么是视觉语言模型，比较了不同的实现方法，并探讨了它们与更广泛的多模态通用人工智能系统的区别。本章还更深入地研究了视觉语言模型，并介绍了基于输出对多模态系统进行分类的方法。

Chapter 2: Deep Dive into Multimodal Systems - This chapter focuses on vision-language models and their role in multimodal AI. It explains what vision-language models are, compares different implementation approaches, and explores how they differ from broader multimodal GenAI systems. The chapter also looks at vision-language models in more depth and introduces ways to classify multimodal systems based on their outputs.

第三章：实现单模态本地GenAI系统——本章探讨构建GenAI系统的实践方面。首先介绍GPU在当今人工智能领域的作用以及如何利用本地GPU。然后介绍Ollama，包括如何使用它生成PDF文档。接下来，解释RAG的工作原理，以及有效实现RAG所面临的关键挑战。

Chapter 3: Implementing Unimodal Local GenAI System - This chapter explores the practical side of building GenAI systems. It begins with the role of GPUs in today’s AI landscape and how to make use of a local GPU. The chapter then introduces Ollama, including how to generate a PDF document with it. Moving forward, it explains how RAG works, along with the key challenges involved in implementing RAG effectively.

第四章：实现基于单模态 API 的 GenAI 系统- 本章将通过实践操作，介绍如何使用 OpenAI 的 API 和模型。它将讲解如何从使用 OpenAI 完成基本任务逐步过渡到构建更高级的智能体 AI 解决方案。您将学习如何执行多文档查询，如何使用 OpenAI 和 Faiss 实现模块化的检索增强型生成系统，并探索一系列扩展这些功能的步骤。

Chapter 4: Implementing Unimodal API-based GenAI Systems - This chapter provides a hands-on introduction to working with OpenAI’s APIs and models. It explains how to move from using OpenAI for basic tasks to building more advanced agentic AI solutions. You will learn how to perform multi-document queries, implement a modular retrieval-augmented generation system using OpenAI and Faiss, and explore a set of to do steps for extending these capabilities further.

第五章：实现人机协同的智能生成人工智能系统——本章重点介绍智能生成人工智能系统的设计和开发。首先阐述此类系统的架构原则，然后详细介绍端到端的人机协同（HITL ）RAG工作流程。接下来，探讨HITL设置如何演进为多智能体HITL RAG系统。最后，本章阐明智能人工智能和人工智能代理之间的区别，重点介绍它们各自的角色和应用。

Chapter 5: Implementing Agentic GenAI Systems with Human-in-the-loop - This chapter focuses on designing and advancing agentic generative AI systems. It starts with principles of architecting such systems and then walks through an end-to-end human-in-the-loop (HITL) RAG workflow. From there, it explores how HITL setups can evolve into multi-agent HITL RAG systems. The chapter concludes by clarifying the differences between agentic AI and AI agents, highlighting their distinct roles and applications..

第六章：两阶段和多阶段GenAI系统 ——本章深入探讨了密集检索系统中交互的概念及其在RAG（评级、可用性、可寻址）中的重要性。它解释了交互模型在两阶段RAG系统中的作用，并比较了不同的重排序策略，包括延迟交互、完全交互和多向量模型。本章随后介绍了两阶段和多阶段RAG架构，讨论了用于评估检索结果的评分机制，并演示了如何实现具有路由的多阶段RAG工作流程，以获得更准确、更高效的响应。

Chapter 6: Two and Multi-stage GenAI Systems - This chapter provides a deep understanding of the concepts of interactions within dense retrieval systems and their importance in RAG. It explains the role of interaction models in two-stage RAG systems and compares different reranking strategies, including late interaction, full interaction, and multi-vector models. The chapter then introduces two-stage and multi-stage RAG architectures, discusses grading mechanisms for evaluating retrieved results, and demonstrates how to implement a multi-stage RAG workflow with routing for more accurate and efficient responses.

第七章：构建双向多模态检索系统 ——本章介绍多模态系统及其基于输出的分类方法。随后，本章阐述多模态检索系统的工作原理，并提供带有详细步骤说明的代码实现。本章最后附有练习题，供读者应用所学知识并加深理解。

Chapter 7: Building a Bidirectional Multimodal Retrieval System - This chapter introduces multimodal systems and how they can be classified based on their outputs. It then explains the working of a multimodal retrieval system and provides a code implementation with step-by-step explanation. The chapter closes with a to do section, giving readers practical exercises to apply and deepen their understanding.

第八章：构建多模态 RAG 系统——本章重点介绍使用 LLM 进行生成和评估的实用方法。首先介绍生成技术的实现，然后介绍 LLM 作为评判者的概念及其在构建推荐系统中的应用。本章还涵盖如何将评分机制与 OpenAI 集成以改进评估。最后，本章附有练习题，供读者将这些理念应用于实践。

Chapter 8: Building a Multimodal RAG System - This chapter focuses on practical approaches to generation and evaluation using LLMs. It begins with the implementation of generation techniques, followed by an introduction to the concept of LLM-as-a-judge and its application in building recommender systems. The chapter also covers how to incorporate grading mechanisms with OpenAI to improve evaluation. It concludes with a to do section, giving readers exercises to apply these ideas in practice.

第九章：基于重排序的GenAI系统构建——本章探讨了重排序的概念及其在改进检索和RAG系统中的关键作用。它解释了重排序如何在基于文本和多模态的环境中应用，重点介绍了如何在多模态RAG中使用交叉编码器。本章还介绍了多模态环境下的交叉编码器架构以及RAG系统中的多索引嵌入思想。除了这些概念之外，本章还提供了带有详细解释的代码实现，并在最后附有练习部分，以帮助读者实践并巩固理解。

Chapter 9: Building GenAI Systems with Reranking - This chapter explores the concept of reranking and its critical role in improving retrieval and RAG systems. It explains how reranking is applied in both text-based and multimodal contexts, with a focus on using cross-encoders in multimodal RAG. The chapter also introduces the cross-encoder architecture in multimodal settings and the idea of multi-index embedding within RAG systems. Alongside these concepts, it provides a code implementation with detailed explanation and concludes with a to do section to help readers practice and solidify their understanding.

第十章：多模态GenAI的检索优化 ——本章探讨如何提高检索系统的效率和效果。首先概述检索系统的常见缺陷，然后介绍各种优化技术来解决这些局限性。本章还详细探讨了检索优化，展示了如何应用这些方法来提升性能。随后，本章重点关注多模态RAG系统，解释了自适应索引刷新如何提高其准确性和响应速度。最后，本章提供了一个练习部分，供读者将这些理念应用于实践。

Chapter 10: Retrieval Optimization for Multimodal GenAI - This chapter examines how to make retrieval systems more efficient and effective. It begins by outlining common drawbacks of retrieval systems, then introduces various optimization techniques to address these limitations. The chapter also explores retrieval optimization in detail, showing how these methods can be applied to improve performance. It then shifts focus to multimodal RAG systems, explaining how adaptive index refresh can enhance their accuracy and responsiveness. Finally, it provides a to do section with exercises for readers to apply these ideas in practice.

第十一章：构建以语音为输入的多模态GenAI系统——本章探讨了RAG如何超越图像和文本的范畴。它介绍了将RAG扩展到其他模态的核心概念，并展示了如何将语音接口集成到RAG架构中。本章还提供了一个支持语音的RAG系统的分步代码实现，演示了如何将这些理念付诸实践。

Chapter 11: Building Multimodal GenAI Systems with Voice as Input - This chapter explores how RAG extends beyond just image and text. It introduces the core concepts of expanding RAG to other modalities and shows how speech interfaces can be integrated into the RAG architecture. The chapter also provides a step-by-step code implementation of a voice-enabled RAG system, demonstrating how to bring these ideas into practice.

第十二章：高级多模态GenAI系统——本章重点阐述推理在GenAI系统中的重要性。它解释了GenAI中使用的不同类型的推理，以及它们对于构建更可靠、更智能的模型为何至关重要。本章还介绍了用于评估AI系统推理能力的关键基准。

Chapter 12: Advanced Multimodal GenAI Systems - This chapter highlights the importance of reasoning in GenAI systems. It explains the different types of reasoning used in GenAI and why they matter for building more reliable and intelligent models. The chapter also introduces key benchmarks that are used to evaluate reasoning capabilities in AI systems.

第十三章：高级多模态GenAI系统实现——本章重点探讨如何通过有效的提示技术增强GenAI的推理能力。随后，本章将探索在不同阶段引入推理的专用架构——首先是在重排序阶段，用于优化结果；其次是在推荐阶段，推理有助于提供更准确、更具上下文感知能力的建议。

Chapter 13: Advanced Multimodal GenAI Systems Implementation - This chapter focuses on how reasoning can be enhanced in GenAI through effective prompting techniques. It then explores specialized architectures that bring reasoning into play at different stages—first during reranking, where results are refined, and then at the recommendation stage, where reasoning helps deliver more accurate and context-aware suggestions.

第十四章：构建文本到 SQL 系统- 本章深入探讨文本到 SQL 的复杂性，以及它为何被认为是一个具有挑战性的问题。本章首先解释基本概念，然后探讨文本到 SQL 能够产生重大影响的实际应用。本章讨论了其中的关键挑战，并提供了设计高效文本到 SQL 系统的实用指导。此外，本章还介绍了使用大型语言模型进行实体提取的方法，重点阐述了如何将其与文本到 SQL 集成以提高性能。最后，本章重点介绍了此类系统如何提高数据可访问性和可读性，同时还介绍了性能指标和最佳实践，以确保可靠性。

Chapter 14: Building Text-to-SQL Systems - This chapter delves into the complexities of text-to-SQL and why it is considered a challenging problem. It begins by explaining the basic concepts and then explores real-world applications where text-to-SQL can make a significant impact. The chapter discusses the key challenges involved, followed by practical guidance on designing an effective text-to-SQL system. It also covers entity extraction using large language models, highlighting how this integrates with text-to-SQL to improve performance. Finally, the chapter emphasizes how such systems can enhance data accessibility and literacy, while also introducing performance metrics and best practices to ensure reliability.

第十五章：智能文本转SQL系统及架构决策——本章介绍专为实时零售智能而设计的智能文本转SQL系统的设计和实现。本章详细解释了系统的架构，并提供了代码演练以帮助读者更好地理解。此外，本章还提供了一个逐步流程图，展示了系统如何处理查询并生成有意义的输出。最后，本章展示了文本转SQL系统生成的实际结果，以及这些结果如何解决最初的问题。

Chapter 15: Agentic Text-to-SQL Systems and Architecture Decision-Making - This chapter presents the design and implementation of an agentic text-to-SQL system tailored for real-time retail intelligence. It explains the system’s architecture in detail, along with code walkthroughs for better understanding. A step-by-step pipeline is provided to show how the system processes queries, leading to meaningful outputs. The chapter concludes by demonstrating the actual results generated by the text-to-SQL system and how they address the original problem statement.

第十六章：利用 GenAI 从图像中提取文本——本章介绍了三种应用 GenAI 进行光学字符识别 (OCR) 的不同方法。它解释了 OCR 如何处理图像，以及如何将其扩展到包含文本、图像和其他元素的多模态文档。本章最后附有练习题，供读者应用和巩固所学知识。

Chapter 16: GenAI for Extracting Text from Images - This chapter introduces three different approaches to applying GenAI for optical character recognition. It explains how OCR works on images, as well as how it can be extended to multimodal documents that combine text, images, and other elements. The chapter concludes with a to do section, giving readers practical exercises to apply and reinforce what they have learned.

第十七章：将传统人工智能/机器学习集成到 GenAI 工作流程中——本章通过一个详细的案例研究，探讨如何将传统机器学习模型集成到 GenAI 工作流程中。本章展示了一个混合集成学习在电信欺诈检测中的实际应用案例，说明了如何将 XGBoost 等模型封装并增强到基于机器学习的系统中。此外，本章还对机器学习模型与 GenAI 结合以创建混合解决方案的不同方法进行了比较概述。最后，本章提供了一个实践练习部分，为读者提供了加深理解的实践活动。

Chapter 17: Integrating Traditional AI/ML into GenAI Workflow - This chapter explores how traditional machine learning models can be integrated into GenAI workflows through a detailed case study. It presents a practical use case of hybrid ensemble learning for telecom fraud detection, showing how models like XGBoost can be wrapped and enhanced within an LLM-powered system. The chapter also provides a comparative overview of different ways ML models can be combined with GenAI to create hybrid solutions. It concludes with a to do section, offering readers hands-on activities to deepen their understanding.

第十八章：LLM运维与GenAI评估技术——本章重点阐述运维在构建和运行生产级GenAI应用过程中的重要性。它比较了LLM和RAG系统的评估方法，介绍了RagOps的概念，并强调了持续监控和可观测性平台的重要性。本章还探讨了图增强型RAG如何改进推荐系统，并对现代软件开发中不同的运维实践进行了比较。最后，本章提供了关于如何设置MLflow以管理实验和部署的实用指南。

Chapter 18: LLM Operations and GenAI Evaluation Techniques - This chapter highlights the importance of operations in building and running production-grade GenAI applications. It compares evaluation methods for LLMs and RAG systems, introduces the concept of RagOps, and emphasizes the need for continuous monitoring and observability platforms. The chapter also explores how graph-enhanced RAG can improve recommendation systems and provides a comparison of different Ops practices in modern software development. Finally, it offers practical guidance on setting up MLflow for managing experiments and deployments.

代码包和彩色图像

Code Bundle and Coloured Images

请点击链接下载

Please follow the link to download the

代码包和本书的彩色图片：

Code Bundle and the Coloured Images of the book:

https://rebrand.ly/78f896

https://rebrand.ly/78f896

本书的代码包也托管在 GitHub 上，地址为https://github.com/bpbpublications/Building-Multimodal-Generative-AI-and-Agentic-Applications。如果代码有任何更新，都会在现有的 GitHub 仓库中更新。

The code bundle for the book is also hosted on GitHub at https://github.com/bpbpublications/Building-Multimodal-Generative-AI-and-Agentic-Applications. In case there’s an update to the code, it will be updated on the existing GitHub repository.

我们丰富的书籍和视频资源库中包含代码包，可在https://github.com/bpbpublications获取。快来看看吧！

We have code bundles from our rich catalogue of books and videos available at https://github.com/bpbpublications. Check them out!

勘误表

Errata

BPB 出版社对我们的工作引以为豪，并遵循最佳实践，以确保内容的准确性，从而为订阅用户提供沉浸式的阅读体验。读者是我们的镜子，我们会利用他们的反馈来反思和改进出版过程中可能出现的人为错误。为了帮助我们保持质量，并及时联系任何可能因意外错误而遇到困难的读者，请写信至：

We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at :

errata@bpbonline.com

errata@bpbonline.com

BPB 出版社全体员工非常感谢您的支持、建议和反馈。

Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family.

加入我们的 Discord 空间

Join our Discord space

加入我们的 Discord 工作区，获取最新资讯、优惠信息、全球科技动态、新版本发布以及与作者的交流：

Join our Discord workspace for latest updates, offers, tech happenings around the world, new releases, and sessions with the authors:

https://discord.bpbonline.com

https://discord.bpbonline.com

目录

Table of Contents

1. 1. 介绍新一代生成式人工智能

   1. 介绍

   2. 结构

   3. 目标

   4. 生成式人工智能概述

   5. 检索系统

      1. 稀疏检索

         1. 密集检索

   6. 发电系统

      1. 发电系统类型

      2. 自回归生成

      3. 提示策略

   7. 了解发电系统的优势所在

      1. 结合检索和生成

   8. 检索增强生成

      1. RAG 工作

      2. 基本 RAG 流水线的架构

      3. RAG架构的类型

      4. 迭代 RAG

      5. 向量数据库和 RAG

      6. RAG 的快速工程

      7. 高级 RAG 技术

      8. RAG的应用

   9. 人工智能系统中的编排

      1. RAG 系统中的编排

      2. 智能体系统中的编排

   10. 人工智能系统中的令牌

   11. 向量数据库

       1. 了解向量数据库

       2. 向量数据库中的索引算法

       3. 向量数据库中的搜索算法

       4. 嵌入和嵌入模型

       5. 向量数据库对 RAG 和智能体系统的重要性

   12. 重新排名

   13. 双编码器与交叉编码器

       1. 用于重排序的交叉编码器

   14. 护栏

       1. 护栏类型

       2. 护栏的安装方法

       3. 没有护栏

       4. 护栏解决方案的行业案例

   15. 代理人

       1. 主动型 RAG 与非主动型 RAG

   16. 模型上下文协议

   17. 结论

2. 1. Introducing New Age Generative AI

   1. Introduction

   2. Structure

   3. Objectives

   4. Overview of generative AI

   5. Retrieval system

      1. Sparse retrieval

         1. Dense retrieval

   6. Generation system

      1. Types of generation systems

      2. Autoregressive generation

      3. Prompting strategies

   7. Understanding where generation systems excel

      1. Combining retrieval and generation

   8. Retrieval-augmented generation

      1. RAG working

      2. Architecture of a basic RAG pipeline

      3. Types of RAG architectures

      4. Iterative RAG

      5. Vector databases and RAG

      6. Prompt engineering for RAG

      7. Advanced RAG techniques

      8. Applications of RAG

   9. Orchestration in AI systems

      1. Orchestration in RAG systems

      2. Orchestration in agentic systems

   10. Tokens in AI systems

   11. Vector database

       1. Understanding vector databases

       2. Indexing algorithms in vector databases

       3. Search algorithms in vector databases

       4. Embeddings and embedding models

       5. Importance of vector databases for RAG and agentic systems

   12. Reranking

   13. Bi-encoders vs. cross-encoders

       1. Cross-encoders for reranking

   14. Guardrails

       1. Types of guardrails

       2. Methods of applying guardrails

       3. Without guardrails

       4. Industry examples of guardrail solutions

   15. Agents

       1. Agentic RAG vs. non-agentic RAG

   16. Model Context Protocols

   17. Conclusion

3. 2. 多式联运系统深度解析

   1. 介绍

   2. 结构

   3. 目标

   4. 理解视觉语言模型

      1. 视觉语言模型的分类

      2. 视觉语言模型的核心架构组件

      3. 视觉语言模型面临的挑战

      4. 多模态GenAI系统

   5. 多模态向量嵌入

      1. 多模态向量数据库

      2. 收藏

      3. 点和点 ID

      4. 向量

      5. 有效载荷

      6. 存储和矢量存储

      7. 索引

   6. 实施方案比较

      1. 单个集合，按有效载荷分区

      2. 具有全局索引的多个集合

   7. 多模态生成式人工智能系统与虚拟语言模型

   8. 视觉语言模型

      1. 多模态生成式人工智能系统

      2. 使用视觉语言模型

      3. 使用多模态生成式人工智能系统

      4. 实际案例比较

   9. 基于输出的多模态系统分类

      1. 文本转图像系统

      2. 图像转文本系统

      3. 文本和图像系统

      4. 纯文本到规格和图像系统

      5. 文本转SQL系统

      6. 文本转代码系统

   10. 结论

4. 2. Deep Dive into Multimodal Systems

   1. Introduction

   2. Structure

   3. Objectives

   4. Understanding vision-language models

      1. Categories of vision-language models

      2. Core architectural components of vision-language models

      3. Challenges in vision-language models

      4. Multimodal GenAI system

   5. Multimodal vector embedding

      1. Multimodal vector database

      2. Collections

      3. Points and point IDs

      4. Vectors

      5. Payload

      6. Storage and vector store

      7. Indexing

   6. Implementation comparisons

      1. Single collection, partitioned via payload

      2. Multiple collections with global indexing

   7. Multimodal generative AI systems vs. VLMs

   8. Vision-language models

      1. Multimodal generative AI systems

      2. Using vision-language models

      3. Using multimodal generative AI systems

      4. Real-world example comparison

   9. Output-based classification of multimodal systems

      1. Text-to-image systems

      2. Image-to-text systems

      3. Text and image systems

      4. Text-only to specifications and image systems

      5. Text-to-SQL systems

      6. Text-to-code systems

   10. Conclusion

5. 3. 实现单模态本地GenAI系统

   1. 介绍

   2. 结构

   3. 目标

   4. GPU 在当今生成式人工智能系统中的应用

   5. 使用本地 GPU

      1. 建筑组件

   6. 关于奥拉玛

      1. Ollama 的替代品

   7. 使用 Ollama 生成 PDF 文档

   8. RAG 实现

      1. 加载并分块 PDF 文档

      2. LangChain 中的替代组块策略

      3. 创建带有元数据的嵌入

         1. 在代码中使用它们

      4. 语义搜索和关键词搜索相结合的混合搜索

         1. 您可以使用的其他寻回犬

      5. 对话记忆缓冲区

      6. LLM配置自然语言生成

      7. ReAct 提示模板

      8. 构建对话式问答链

      9. 用户聊天循环

   9. RAG面临的挑战

   10. 结论

6. 3. Implementing Unimodal Local GenAI System

   1. Introduction

   2. Structure

   3. Objectives

   4. GPU in today’s generative AI systems

   5. Using a local GPU

      1. Architectural components

   6. About Ollama

      1. Alternatives to Ollama

   7. Generate a PDF document with Ollama

   8. RAG implementation

      1. Load and chunk the PDF document

      2. Alternative chunking strategies in LangChain

      3. Creating embeddings with metadata

         1. Using them in code

      4. Hybrid search with semantic and keyword

         1. Other retrievers you can use

      5. Conversation memory buffer

      6. LLM configuration natural language generation

      7. ReAct prompt template

      8. Building the conversational QA chain

      9. User chat loop

   9. Challenges in RAG

   10. Conclusion

7. 4. 实现基于单模态 API 的 GenAI 系统

   1. 介绍

   2. 结构

   3. 目标

   4. OpenAI API 和模型入门

      1. OpenAI 公司

      2. OpenAI API 概述

   5. 核心 API 端点

      1. OpenAI 主要模型

      2. 访问 OpenAI 模型

      3. 选择合适的型号

      4. 初学者最佳实践

      5. 从 OpenAI 到智能体人工智能

      6. OpenAI 的智能体 API 生态系统

         1. 响应 API

         2. 代理 SDK

         3. 操作员

         4. 法典

         5. 助手 API

   6. 多文档查询

   7. 使用 OpenAI 实现模块化 RAG

      1. 主控制器

      2. 配置

      3. 嵌入初始化

      4. 矢量图库设置

      5. 元数据标记

      6. 文档加载和分块

      7. 杂交寻回犬

         1. 在检索过程中强制执行基于元数据的过滤

      8. 语言模型

      9. 提示模板

      10. RAG 链组件

      11. 会话记忆

      12. 依赖关系

   8. 待办事项

   9. 结论

8. 4. Implementing Unimodal API-based GenAI Systems

   1. Introduction

   2. Structure

   3. Objectives

   4. Getting started with OpenAI APIs and models

      1. OpenAI as a company

      2. Overview of the OpenAI API

   5. Core API endpoints

      1. Major OpenAI models

      2. Accessing OpenAI models

      3. Choosing the right model

      4. Best practices for beginners

      5. From OpenAI to agentic AI

      6. OpenAI’s agentic API ecosystem

         1. Responses API

         2. Agents SDK

         3. Operator

         4. Codex

         5. Assistants API

   6. Multi-document query

   7. Implementing modular RAG with OpenAI

      1. Main controller

      2. Configuration

      3. Embedding initialization

      4. Vector store setup

      5. Metadata tagging

      6. Document loading and chunking

      7. Hybrid retriever

         1. Enforce metadata-based filtering during retrieval

      8. Language model

      9. Prompt template

      10. RAG chain assembly

      11. Conversational memory

      12. Dependencies

   8. To do

   9. Conclusion

9. 5. 实现具有人机交互的智能基因人工智能系统

   1. 介绍

   2. 结构

   3. 目标

   4. 构建智能体GenAI系统

      1. 平行模式

      2. 序列模式

      3. 循环图案

      4. 路由器图案

      5. 聚合器模式

      6. 网络模式

      7. 层级模式

      8. 人机交互模式

      9. 共享工具模式

      10. 带有工具模式的数据库

      11. 使用工具进行内存转换

      12. 规划者-执行者模式

      13. 批评者或验证者模式

      14. 谈判者模式

      15. 多模态代理模式

      16. 投票或共识模式

      17. 主管-下属模式

      18. 监视或恢复模式

      19. 时间规划模式

          1. 人机交互

   5. 端到端的人机交互 RAG 工作流程

   6. 从 HITL 到多智能体人机交互 RAG

   7. 智能体人工智能与人工智能代理

   8. 结论

10. 5. Implementing Agentic GenAI Systems with Human-in-the-loop

    1. Introduction

    2. Structure

    3. Objectives

    4. Architecting agentic GenAI systems

       1. Parallel pattern

       2. Sequential pattern

       3. Loop pattern

       4. Router pattern

       5. Aggregator pattern

       6. Network pattern

       7. Hierarchical pattern

       8. Human-in-the-loop pattern

       9. Shared tools pattern

       10. Database with tools pattern

       11. Memory transformation using tools

       12. Planner-executor pattern

       13. Critic or validator pattern

       14. Negotiator pattern

       15. Multimodal agent pattern

       16. Voting or consensus pattern

       17. Supervisor-subordinate pattern

       18. Watchdog or recovery pattern

       19. Temporal planner pattern

           1. Human-in-the-loop

    5. End-to-end human-in-the-loop RAG workflow

    6. From HITL to multi-agent human-in-the-loop RAG

    7. Agentic AI vs. AI agents

    8. Conclusion

11. 6. 两阶段和多阶段GenAI系统

    1. 介绍

    2. 结构

    3. 目标

    4. 密集检索中的交互概念

       1. 无互动

       2. 完全互动

       3. 后期互动

       4. 多向量表示

       5. 与后期交互架构的差异

    5. 交互模型在两阶段 RAG 系统中的作用

       1. 检索阶段的互动

    6. 使用各种交互模型进行重排序

       1. 集成到两阶段 RAG 架构中

    7. 两阶段 RAG 架构

       1. 第一阶段密集检索

       2. 第二阶段，语义精确度重排序

       3. 两阶段设计的战略作用

       4. 两阶段 RAG 与后期互动

          1. ColBERT 和 ColPali 的功能

          2. 使用两阶段 RAG

    8. 多阶段 RAG

       1. 超越两阶段系统

       2. 多阶段 RAG 的组成部分

       3. 多阶段 RAG 的优势

          1. 多阶段 RAG 的类型

    9. 评分机制

       1. 挑战与考量

       2. 多阶段 RAG 系统中的代币利用率

       3. 评分类型

    10. 实现多阶段 RAG 工作流程及路由

    11. 结论

12. 6. Two and Multi-stage GenAI Systems

    1. Introduction

    2. Structure

    3. Objectives

    4. Concepts of interactions in dense retrievals

       1. No interaction

       2. Full interaction

       3. Late interaction

       4. Multi-vector representations

       5. Differentiation from late interaction architectures

    5. Role of interaction models in two-stage RAG systems

       1. Interaction in the retrieval phase

    6. Reranking with various interaction models

       1. Integration into two-stage RAG architectures

    7. Two-stage RAG architecture

       1. Stage one dense retrievals

       2. Stage-two, reranking for semantic precision

       3. The strategic role of two-stage design

       4. Two-stage RAG vs. late interaction

          1. Capabilities of ColBERT and ColPali

          2. Use of two-stage RAG

    8. Multi-stage RAG

       1. Beyond two-stage systems

       2. Components of multi-stage RAG

       3. Benefits of multi-stage RAG

          1. Types of multi-stage RAG

    9. Grading mechanisms

       1. Challenges and considerations

       2. Token utilization in multi-stage RAG systems

       3. Grading types

    10. Implementation of multi-stage RAG workflow with routing

    11. Conclusion

13. 7. 构建双向多模态检索系统

    1. 介绍

    2. 结构

    3. 目标

    4. 基于输出的多模态系统分类

       1. 集成和设计影响

    5. 理解多模态检索系统

       1. 技术架构

          1. 应用及影响

    6. 代码实现及说明

       1. 要求

          1. 前端

          2. 数据目录

          3. 检索系统

          4. 装载机

       2. 嵌入工具

          1. 索引构建器

          2. 运行整个代码的过程

    7. 为了读者们

    8. 结论

14. 7. Building a Bidirectional Multimodal Retrieval System

    1. Introduction

    2. Structure

    3. Objectives

    4. Output-based classification of multimodal systems

       1. Integration and design implications

    5. Understanding a multimodal retrieval system

       1. Technical architecture

          1. Applications and implications

    6. Code implementation and explanation

       1. Requirement

          1. Frontend

          2. Data directory

          3. The retrieval system

          4. Loaders

       2. Embedding utils

          1. Index builder

          2. Process to run the entire code

    7. To do for the readers

    8. Conclusion

15. 8. 构建多模式 RAG 系统

    1. 介绍

    2. 结构

    3. 目标

    4. 生成过程的实现

       1. 架构组件和工作流程

          1. 发电机

    5. 基于多模态LLM的推荐系统

       1. 领先架构及示例

    6. 将评分与 OpenAI 结合使用

       1. 导入语句

       2. 生成式响应评分器

       3. 检索相关性评分器

       4. 分级和生成模型

    7. 用于评分的云端LLM

       1. 法学硕士担任法官

       2. 原理和功能

    8. 待办事项

    9. 结论

16. 8. Building a Multimodal RAG System

    1. Introduction

    2. Structure

    3. Objectives

    4. Implementation of generation

       1. Architectural components and workflow

          1. Generator

    5. Multimodal LLM-based recommender system

       1. Leading architectures and examples

    6. Incorporate grading with OpenAI

       1. Import statements

       2. Generative responsive grader

       3. Retrieval relevance grader

       4. Grading and generation models

    7. Cloud LLMs for grading

       1. LLM-as-a-judge

       2. Rationale and functionality

    8. To do

    9. Conclusion

17. 9. 利用重排序构建 GenAI 系统

    1. 介绍

    2. 结构

    3. 目标

    4. 重新排名

    5. 信息检索和RAG系统中的重排序

       1. RAG管道中的重新排序

    6. 基于交叉编码器的多模态RAG重排序

    7. 多模态环境下的交叉编码器架构

       1. 交叉编码器与后期交互重排序器

       2. 多模态检索中的应用

       3. 商业排名器

       4. 交叉编码器概述

          1. 交叉编码器及其在嵌入中的作用

    8. RAG系统中的多索引嵌入

    9. 代码实现及说明

    10. 待办事项

        1. 安装说明

    11. 结论

18. 9. Building GenAI Systems with Reranking

    1. Introduction

    2. Structure

    3. Objectives

    4. Reranking

    5. Reranking in information retrieval and RAG systems

       1. Reranking in RAG pipelines

    6. Reranking using cross-encoder in multimodal RAG

    7. Cross-encoder architecture in multimodal settings

       1. Cross-encoders vs. late interaction rerankers

       2. Applications in multimodal retrieval

       3. Commercial reranker

       4. Recap of cross-encoder

          1. Cross-encoders and their role in embedding

    8. Multi-index embedding in RAG systems

    9. Code implementation and explanation

    10. To do

        1. Setup instructions

    11. Conclusion

19. 10. 多模态 GenAI 的检索优化

    1. 介绍

    2. 结构

    3. 目标

    4. 检索优化技术

    5. 回收系统的缺点

    6. 检索优化技术缓解了这些限制

       1. 多索引嵌入

       2. 基于模态的多模态查询路由

       3. 查询扩展

       4. 嵌入归一化

       5. 混合检索

          1. 分数标准化

       6. 使用交叉编码器进行重排序

          1. 预滤波阈值

       7. 自适应索引刷新

       8. 检索优化技术

          1. GA实现用于优化模态

          2. 解释

       9. 具有自适应索引刷新的多模态 RAG 系统

       10. 一次性或计划性索引刷新脚本

    7. 利用自适应刷新增强多模态 RAG

       1. Qdrant中的向量嵌入管道和存储

       2. 两阶段检索和多向量重排序

       3. 上下文组装和语言生成

       4. 自适应嵌入刷新机制

       5. 索引行为

    8. 待办事项

    9. 结论

20. 10. Retrieval Optimization for Multimodal GenAI

    1. Introduction

    2. Structure

    3. Objectives

    4. Retrieval optimization techniques

    5. Drawbacks retrieval systems

    6. Retrieval optimization techniques mitigating the limitations

       1. Multi-index embedding

       2. Modality-based routing for multimodal queries

       3. Query expansion

       4. Embedding normalization

       5. Hybrid retrieval

          1. Score normalization

       6. Reranking with cross-encoders

          1. Prefiltering thresholds

       7. Adaptive index refresh

       8. Retrieval optimization techniques

          1. GA implementation for optimizing modality

          2. Explanation

       9. Multimodal RAG system with adaptive index refresh

       10. One-time or scheduled index refresh script

    7. Enhancing multimodal RAG with adaptive refresh

       1. Vector embedding pipeline and storage in Qdrant

       2. Two-stage retrieval and multi-vector reranking

       3. Context assembly and language generation

       4. Adaptive embedding refresh mechanism

       5. Indexing behavior

    8. To do

    9. Conclusion

21. 11. 构建以语音为输入的多模态GenAI系统

    1. 介绍

    2. 结构

    3. 目标

    4. RAG 超越图像和文本 RAG

    5. 概念

    6. 将语音接口集成到 RAG 架构中

    7. 语音驱动的 RAG 系统的代码实现

       1. 技术栈概述

       2. 前端

       3. 主语音管道

    8. 结论

22. 11. Building Multimodal GenAI Systems with Voice as Input

    1. Introduction

    2. Structure

    3. Objectives

    4. RAG beyond image and text RAG

    5. Concepts

    6. Integrating speech interfaces into RAG architecture

    7. Code implementation of the voice-enabled RAG system

       1. Tech stack overview

       2. Frontend

       3. Main voice-enabled pipeline

    8. Conclusion

23. 12. 先进的多模态基因人工智能系统

    1. 介绍

    2. 结构

    3. 目标

    4. 推理在生成式人工智能系统中的关键作用

       1. 从世代到审议

       2. 人工智能系统中的信任和可解释性

       3. 处理歧义和消除歧义

       4. 多模式整合需要逻辑组合

       5. 快捷工程和CoT推理

       6. 重新排序和元推理

       7. 学习可推广的策略

       8. 人机协作

       9. 智能人工智能基金会

    5. GenAI中的推理及其类型

       1. 人工智能中的演绎推理

       2. 人工智能中的归纳推理

       3. 人工智能中的溯因推理

       4. 人工智能中的类比推理

       5. 常识推理

       6. 因果推理

       7. 空间推理

       8. 时间推理

       9. 数学推理

       10. 基于工具的推理和ReAct代理

       11. 人工智能系统中的多模态推理与融合

    6. 关于推理基准

    7. 结论

24. 12. Advanced Multimodal GenAI Systems

    1. Introduction

    2. Structure

    3. Objectives

    4. The critical role of reasoning in generative AI systems

       1. From generation to deliberation

       2. Trust and explainability in AI systems

       3. Handling ambiguity and disambiguation

       4. Multimodal integration requires logical composition

       5. Prompt engineering and CoT reasoning

       6. Reranking and meta-reasoning

       7. Learning generalizable strategies

       8. Human-AI collaboration

       9. Foundation for agentic AI

    5. Reasoning in GenAI and their types

       1. Deductive reasoning in AI

       2. Inductive reasoning in AI

       3. Abductive reasoning in AI

       4. Analogical reasoning in AI

       5. Commonsense reasoning

       6. Causal reasoning

       7. Spatial reasoning

       8. Temporal reasoning

       9. Mathematical reasoning

       10. Tool-based reasoning and ReAct agents

       11. Multimodal reasoning and fusion in AI systems

    6. About reasoning benchmark

    7. Conclusion

25. 13. 高级多模态GenAI系统实现

    1. 介绍

    2. 结构

    3. 目标

    4. 基因人工智能系统中推理的提示技术

       1. 基本提示技巧

          1. 零次提示

          2. 少镜头提示

       2. 基因人工智能系统中推理的高级提示策略

    5. 重排序阶段推理的架构

       1. 模块：loaders.py

       2. 模块：embedding_utils.py

       3. 模块：index_builder.py

       4. 模块：reranker.py

       5. 模块：langgraph_agent.py

          1. langgraph_agent.py 模块的代理特性

          2. 代理属性和功能

    6. 推荐阶段推理的架构

       1. 数据集

       2. 推荐引擎的目标

          1. 最终检索限制

       3. 模块化代码库分解

    7. 结论

26. 13. Advanced Multimodal GenAI Systems Implementation

    1. Introduction

    2. Structure

    3. Objectives

    4. Prompting techniques for reasoning in GenAI systems

       1. Basic prompting techniques

          1. Zero-shot prompting

          2. Few-shot prompting

       2. Advanced prompting strategies for reasoning in GenAI systems

    5. Architecture for reasoning at the reranking stage

       1. Module: loaders.py

       2. Module: embedding_utils.py

       3. Module: index_builder.py

       4. Module: reranker.py

       5. Module: langgraph_agent.py

          1. Agentic characteristics of the langgraph_agent.py module

          2. Agentic attributes and functionality

    6. Architecture for reasoning at the recommendation stage

       1. The dataset

       2. Goal of the recommendation engine

          1. Final retrieval constraints

       3. Modular codebase breakdown

    7. Conclusion

27. 14. 构建文本到 SQL 系统

    1. 介绍

    2. 结构

    3. 目标

    4. 文本转SQL是一个难题

    5. 理解基本概念

    6. 探索实际应用

    7. 主要挑战

    8. 关于设计文本到 SQL 系统的实用指南

    9. 使用LLM和文本到SQL系统的实体提取

       1. 架构概述

    10. 提高数据可访问性和可读性

    11. 绩效指标和最佳实践

        1. 精确匹配准确度

        2. 执行准确率

        3. 组件级精度

        4. 查询执行成功率

        5. 语义等价和规范化

        6. 人为评估

        7. 延迟和吞吐量指标

        8. 绩效评估最佳实践

    12. 结论

28. 14. Building Text-to-SQL Systems

    1. Introduction

    2. Structure

    3. Objectives

    4. Text-to-SQL a hard problem

    5. Understanding basic concepts

    6. Exploration of real-world applications

    7. Key challenges

    8. Practical guidance on designing a text-to-SQL system

    9. Entity extraction using LLM and text-to-SQL system

       1. Architecture overview

    10. Enhance data accessibility and literacy

    11. Performance metrics and best practices

        1. Exact match accuracy

        2. Execution accuracy

        3. Component-level accuracy

        4. Query execution success rate

        5. Semantic equivalence and canonicalization

        6. Human evaluation

        7. Latency and throughput metrics

        8. Best practices for performance evaluation

    12. Conclusion

29. 15. 智能体文本到 SQL 系统及架构决策

    1. 介绍

    2. 结构

    3. 目标

    4. 用于实时零售情报的代理式文本到 SQL 系统

       1. 业务挑战和问题陈述

    5. 文本到 SQL 系统的架构和代码说明

    6. 逐步管道说明

       1. 文件夹结构

          1. 要求

          2. 安装说明

       2. 理解每个 Python 脚本

          1. 主执行层

          2. 代理模块

          3. 核心基础设施层

          4. 面向任务的模块

          5. 前端界面

          6. 系统设置和索引初始化

       3. 代码的内部运作

          1. 代理和工具概述

    7. 文本转 SQL 系统的输出

       1. 实体和数据库详细摘要

          1. 生成的 SQL 查询

          2. SQL 查询等级

          3. 总结等级

    8. 针对初始问题陈述的解决方案

    9. 结论

30. 15. Agentic Text-to-SQL Systems and Architecture Decision-Making

    1. Introduction

    2. Structure

    3. Objectives

    4. Agentic text-to-SQL system for real-time retail intelligence

       1. Business challenge and problem statement

    5. Architecture and code explanation of text-to-SQL system

    6. Step-by-step pipeline explanation

       1. Folder structure

          1. Requirements

          2. Setup instructions

       2. Understanding each Python script

          1. Main execution layer

          2. Agent modules

          3. Core infrastructure layer

          4. Task-oriented modules

          5. Frontend interface

          6. System setup and index initialization

       3. Inner workings of the code

          1. Agent and tool summary

    7. Output from the text-to-SQL system

       1. Detailed entity and database summary

          1. Generated SQL query

          2. SQL query grade

          3. Summary Grade

    8. Solution to the initial problem statement

    9. Conclusion

31. 16. GenAI 从图像中提取文本

    1. 介绍

    2. 结构

    3. 目标

    4. 基于GenAI的OCR的三种方法

       1. 购物助手用例

    5. 图像OCR识别

       1. 购物协助

       2. 架构概述

       3. 了解输出

    6. 对多模态文档进行光学字符识别

       1. Mistral 的 OCR

       2. 上下文中的正则表达式

       3. 收据数据中的 OCR 功能

    7. 待办事项

    8. 结论

32. 16. GenAI for Extracting Text from Images

    1. Introduction

    2. Structure

    3. Objectives

    4. Three approaches to GenAI-based OCR

       1. Shopping assistance use case

    5. OCR on image

       1. Building shopping assistance

       2. Architecture overview

       3. Understanding the output

    6. OCR on a multimodal document

       1. Mistral's OCR

       2. The regex in context

       3. OCR in receipt data

    7. To do

    8. Conclusion

33. 17. 将传统人工智能/机器学习集成到 GenAI 工作流程中

    1. 介绍

    2. 结构

    3. 目标

    4. 案例研究

    5. 将传统模型与世代人工智能相结合

       1. 这些混合系统的初始化

    6. 用例

       1. 数据特征和预处理

       2. 基线模型开发与评估

       3. 堆叠式集成学习方法

       4. 在此设置中 LLM 的作用

    7. 将 XG 提升模型封装到 LLM 中

       1. 运行顺序

       2. 代码实现

       3. 模型训练流程

       4. FastAPI 服务层

       5. 用于 FastAPI 推理的工具封装

       6. LangChain 工具注册

       7. 通过 Ollama 与 Mistral 进行代理编排

    8. GenAI工作流程中机器学习模型集成的比较概述

    9. 待办事项

    10. 结论

34. 17. Integrating Traditional AI/ML into GenAI Workflow

    1. Introduction

    2. Structure

    3. Objectives

    4. Case study

    5. Integrating the traditional model with GenAI

       1. Initialization of these hybrid systems

    6. Use case

       1. Data characteristics and preprocessing

       2. Baseline model development and evaluation

       3. Stacked ensemble learning approach

       4. Purpose of the LLM in this setup

    7. Wrapping XG boost model into LLM

       1. Run order

       2. Code implementation

       3. Model training pipeline

       4. FastAPI serving layer

       5. Tool wrapper for FastAPI inference

       6. LangChain tool registration

       7. Agent orchestration with Mistral via Ollama

    8. Comparative overview of ML model integration in GenAI workflows

    9. To do

    10. Conclusion

35. 18. LLM 操作和 GenAI 评估技术

    1. 介绍

    2. 结构

    3. 目标

    4. 生产级 GenAI 应用中运维的重要性

    5. 比较LLM和RAG评估

       1. LLM评估

       2. RAG 评估

          1. 区分的重要性

          2. 评估是GenAI运维的核心

          3. 确保大规模输出质量

       3. 监测漂移和幻觉

       4. 评估检索质量以进行预先调试

       5. 支持版本控制和可追溯性

       6. 反馈回路和自愈系统

    6. RAGOps

       1. 在开发过程中

          1. 开发过程中 RAGOps 的识别

          2. 在 RAGOps 开发过程中进行基准测试

       2. 后期开发

          1. 识别后期发展

          2. RAG系统开发后的基准测试

    7. 持续监测

       1. 实时 RAG 系统中的持续监控

       2. RAGOps 中需要监控的关键指标

       3. 连续监测的技术和工具

       4. 警报、仪表盘和异常检测

       5. 反馈回路和自愈系统

    8. 可观测性平台

       1. 核心可观测性平台

       2. RAG专用评估库

       3. 辅助工具和生态系统集成

    9. 基于图增强的 RAG 推荐系统

       1. 数据摄取管道

       2. 检索和推荐流程

          1. 系统中的智能 RAG 设计和多工具检索

          2. 代理控制回路

          3. 三种互补的检索工具

          4. 代理人的运作角色

          5. 运营风险分析和监控指标

    10. 现代软件开发中各种操作的比较

    11. 安装 MLflow

        1. 可观测性管道

        2. 方法一

        3. 方法二

        4. 使用本地文件系统结构排查 MLflow 故障

    12. 结论

36. 18. LLM Operations and GenAI Evaluation Techniques

    1. Introduction

    2. Structure

    3. Objectives

    4. Importance of Ops in production-grade GenAI applications

    5. Comparing LLM and RAG evaluations

       1. LLM evaluation

       2. RAG evaluation

          1. Importance of distinction

          2. Evaluation as the core of GenAI Ops

          3. Ensuring output quality at scale

       3. Monitoring drift and hallucinations

       4. Evaluating retrieval quality for preemptive debugging

       5. Supporting version control and traceability

       6. Feedback loops and self-healing systems

    6. RAGOps

       1. During development

          1. Identification in RAGOps during development

          2. Benchmarking in RAGOps during development

       2. Post-development

          1. Identify post-development

          2. Benchmarking in RAG systems post-development

    7. Continuous monitoring

       1. Continuous monitoring in live RAG systems

       2. Key metrics to monitor in RAGOps

       3. Techniques and tools for continuous monitoring

       4. Alerting, dashboards, and anomaly detection

       5. Feedback loop and self-healing systems

    8. Observability platforms

       1. Core observability platforms

       2. RAG-specific evaluation libraries

       3. Auxiliary tools and ecosystem integrations

    9. Graph-enhanced RAG-based recommendation system

       1. Data ingestion pipeline

       2. Retrieval and recommendation pipeline

          1. Agentic RAG design and multi-tool retrieval in the system

          2. Agentic control loop

          3. Three complementary retrieval tools

          4. Operational role of the agent

          5. Operational risk analysis and monitoring metrics

    10. Comparison of various Ops in modern software development

    11. Installation of MLflow

        1. Observability pipeline

        2. Approach 1

        3. Approach 2

        4. Troubleshooting MLflow using local filesystem structure

    12. Conclusion

37. 指数

38. Index

介绍

Introduction

本章通过介绍基本概念和基础技术，为掌握新一代生成式人工智能（GenAI ）系统奠定基础。我们首先探讨检索系统和生成系统的区别，然后深入研究向量数据库、搜索算法、嵌入技术、索引和重排序，这些都是构建智能高效人工智能解决方案的关键要素。此外，我们还讨论了反射和防护机制等关键可靠性机制，以确保输出结果的稳健性并与用户意图保持一致。

This chapter sets the stage for mastering new age generative AI (GenAI) systems by introducing essential concepts and foundational technologies. We begin by exploring the difference between retrieval systems and generation systems, followed by an in-depth look at vector databases, search algorithms, embedding techniques, indexing, and reranking, all critical for building intelligent, efficient AI solutions. Key reliability mechanisms, such as reflection and guardrails, are discussed to ensure outputs remain robust and aligned with user intent.

接下来，我们将深入探讨诸如思维链（CoT ）等高级提示方法，以引导人工智能模型进行结构化推理。本章随后转向智能体人工智能，涵盖智能体、工具、推理、规划和动作执行，并扩展到能够执行复杂协作任务的多智能体系统的设计。本章还提供了大型语言模型（LLM ）、大型视觉模型（LVM ）和新兴的大型动作模型（LAM ）的比较概述，以及关于本地模型部署和图形处理器（GPU ）基础设施规划的实用见解。

We then dive into advanced prompting methods like chain of thought (CoT) to guide AI models through structured reasoning processes. Moving into agentic AI, the chapter covers agents, tools, reasoning, planning, and action execution, expanding into the design of multi-agent systems capable of complex, collaborative tasks. A comparative overview of large language models (LLMs), large vision models (LVMs), and emerging large action models (LAMs) is provided, along with practical insights into local model deployment and graphics processing unit (GPU) infrastructure planning.

此外，我们还介绍了语音技术，包括自动语音识别（ASR ）和语音生成，并阐述了内存管理在基于代理的架构中的关键作用。最后，我们介绍了模型上下文协议（MCP ）等行业标准，并区分了GenAI开发人员和GenAI工程师不断变化的职责，为读者进行高级系统设计做好准备。

Further, we introduce speech technologies, including automated speech recognition (ASR) and generation, and explain the critical role of memory management in agent-based architectures. Finally, we present industry standards like Model Context Protocol (MCP) and differentiate the evolving responsibilities of a GenAI developer vs. a GenAI engineer, preparing readers for advanced system design.

结构

Structure

本章涵盖以下主题：

This chapter covers the following topics:

• 生成式人工智能概述
• Overview of generative AI
• 检索系统
• Retrieval system
• 发电系统
• Generation systems
• 了解发电系统的优势所在
• Understanding where generation systems excel
• 检索增强生成
• Retrieval-augmented generation
• 人工智能系统中的编排
• Orchestration in AI systems
• 人工智能系统中的令牌
• Tokens in AI systems
• 向量数据库
• Vector database
• 重新排名
• Reranking
• 双编码器与交叉编码器
• Bi-encoders vs. cross-encoders
• 护栏
• Guardrails
• 代理人
• Agents
• 模型上下文协议
• Model Context Protocols

目标

Objectives

本章旨在帮助读者全面理解设计和部署现代全智能（GenAI）系统所需的关键构建模块。通过探讨检索和生成系统、向量数据库、嵌入技术、高级提示策略、智能体架构和多智能体协作等概念，读者将为构建智能、可扩展的人工智能解决方案奠定坚实的基础。此外，本章还介绍了本地模型部署、GPU 基础设施、语音处理、智能体内存管理以及 MCP 等行业标准等关键主题。这些基础要素对于构建多模态、可靠且可用于生产环境的人工智能应用至关重要。

This chapter aims to equip readers with a comprehensive understanding of the key building blocks essential for designing and deploying modern GenAI systems. By exploring concepts such as retrieval and generation systems, vector databases, embedding techniques, advanced prompting strategies, agentic architectures, and multi-agent collaboration, readers will gain a strong foundation for building intelligent, scalable AI solutions. Additionally, the chapter introduces critical topics like local model deployment, GPU infrastructure, speech processing, memory management in agents, and industry standards like MCPs. These foundational elements are crucial for advancing toward multimodal, reliable, and production-ready AI applications.

生成式人工智能概述

Overview of generative AI

生成模型的演进代表了人工智能领域最重要的范式转变之一。在预训练Transformer （GPT ）出现之前的时代，生成人工智能（GenAI）的发展主要得益于玻尔兹曼机、变分自编码器（VAE ）、生成对抗网络（GAN ）和自编码器等强大技术。这些模型通过生成图像、音频甚至文本等非结构化数据，取得了突破性的成果。例如，GAN彻底革新了逼真图像合成，而VAE则实现了复杂数据空间（包括语音和文档生成）的概率生成建模。

The evolution of generative models represents one of the most significant paradigm shifts in AI. In the pre-generative pre-trained transformers (GPTs) era, GenAI was shaped by powerful techniques such as Boltzmann machines, variational autoencoders (VAEs), generative adversarial networks (GANs), and autoencoders. These models achieved groundbreaking results by generating unstructured data like images, audio, and even text. For instance, GANs revolutionized realistic image synthesis, while VAEs enabled probabilistic generative modeling of complex data spaces, including speech and document generation.

尽管这些早期系统令人印象深刻，但它们通常专注于单一领域的生成，推理、规划或跨任务泛化的能力有限。它们缺乏现代人工智能体验所具备的丰富的上下文理解能力、动态推理能力和任务驱动的灵活性。

While impressive, these earlier systems generally focused on single-domain generation with limited ability to reason, plan, or generalize across tasks. They lacked the rich contextual understanding, dynamic reasoning, and task-driven flexibility that define modern AI experiences.

真正的范式转变并非直接源于 GPT 模型，而是源于 2017 年 Transformer 架构本身的引入（Vaswani 等人在开创性论文《Attention Is All You Need 》中提出）。Transformer 引入了自注意力、并行处理和位置编码的概念，使得模型在规模和能力上都能大规模扩展，远远超出了基于传统循环神经网络( RNN )、长短期记忆网络( LSTM ) 或卷积神经网络( CNN ) 的生成模型的局限性。

The true paradigm shift occurred not directly with GPT models, but with the introduction of the transformer architecture itself in 2017 (in the seminal paper Attention Is All You Need by Vaswani et al.). The transformer introduced the concepts of self-attention, parallel processing, and positional encoding, enabling models to scale massively in both size and capability, far beyond the limits of traditional recurrent neural networks (RNNs), long short-term memories (LSTMs), or convolutional neural networks (CNNs) based generative models.

在Transformer模型的基础上，GPT开启了开放式生成模型的时代，这些模型不仅能够重现数据，还能执行对话、推理、摘要、代码生成和多模态合成等任务。现代GenAI系统如今展现出语义感知、动态问题解决能力以及跨文本、图像和语音的多模态理解能力。

Building on the transformer foundation, GPTs ushered in the era of open-ended generation models capable of not just recreating data but performing tasks like conversation, reasoning, summarization, code generation, and multimodal synthesis. The modern GenAI systems now exhibit semantic awareness, dynamic problem-solving, and multimodal understanding across text, images, and speech.

新时代以以下几项关键进步为标志：

Several key advancements define this new age, which are as follows:

• 在多样化、异构的数据集上进行大规模预训练。
• Massive pre-training on diverse, heterogeneous datasets.
• 扩展规律表明，随着参数、数据和计算量的增加，改进效果可预测。
• Scaling laws showing predictable improvements with more parameters, data, and compute.
• CoT提示技术在引导推理中的应用。
• CoT prompting techniques for guided reasoning.
• 智能体人工智能架构，其中模型不仅可以生成数据，还可以推理、计划和行动。
• Agentic AI architectures where models not only generate but also reason, plan, and act.
• 多智能体系统协作实现复杂目标。
• Multi-agent systems collaborating toward complex goals.
• 跨文本、视觉和音频模态的多模态生成。
• Multimodal generation across text, vision, and audio modalities.
• GPU基础设施的改进和高效模型推动了私有和本地部署的发展。
• Private and local deployments driven by improvements in GPU infrastructure and efficient models.

本书超越了传统的人工智能生成方式，着重探讨如何设计、构建和部署面向推理、规划和行动的全息人工智能（GenAI）——这些系统正在改变着各行各业、企业以及人们的日常生活。理解这一转变至关重要：最初只是数据模仿，如今已发展成为能够增强和自动化人类思维的智能多模态代理。

In this book, we move beyond classical generation, focusing on designing, building, and deploying reasoning, planning, and action-oriented GenAI—the systems that are now transforming industries, enterprises, and everyday experiences. Understanding this transition is key: what started as data mimicry has evolved into intelligent, multimodal agents capable of augmenting and automating human thought itself.

虽然生成模型已经发展到能够创造出丰富且类人的输出，但并非所有人工智能解决方案都仅仅依赖于生成。事实上，当今许多最强大的人工智能系统都将检索与生成相结合，以使输出结果与现实世界的信息紧密相连，从而提高可靠性并减少虚假信息。

While generative models have evolved to create rich, human-like outputs, not all AI solutions rely solely on generation. In fact, many of the most powerful AI systems today combine retrieval with generation to ground their outputs in real-world information, improve reliability, and reduce hallucinations.

在探讨生成策略之前，首先必须了解检索系统，它是人工智能查找、筛选并将相关知识引入对话的核心。检索是现代人工智能基础设施的关键支柱，支持从搜索引擎和推荐系统到高级检索增强生成（RAG ）流程等各种任务。

Before exploring generation strategies, it is essential to first understand retrieval systems, the backbone of how AI finds, filters, and brings relevant knowledge into the conversation. Retrieval forms a critical pillar of modern AI infrastructure, supporting tasks ranging from search engines and recommendation systems to advanced retrieval-augmented generation (RAG) pipelines.

在下一节中，我们将探讨什么是检索系统，它们与纯粹的生成模型有何不同，以及为什么它们对于构建准确、可扩展和生产级的 AI 应用程序来说是必不可少的。

In the next section, we will explore what retrieval systems are, how they differ from pure generative models, and why they are indispensable for building accurate, scalable, and production-grade AI applications.

检索系统

Retrieval system

如今，人工智能系统因其创造力和推理能力而备受赞誉，但许多智能行为的背后都建立在强大的检索机制之上。检索机制通常是人工智能的隐形引擎，它使人工智能能够将输出结果与现实世界的知识联系起来，找到相关事实，并在对话或任务中保持逻辑一致性。要真正理解检索机制如何成为现代人工智能的关键支柱，首先需要了解它的演变历程，从简单的关键词匹配到复杂、学习驱动和记忆增强的技术。

GenAI systems today are celebrated for their creativity and reasoning abilities, but behind many of these intelligent behaviors lies a strong foundation built on retrieval mechanisms. Retrieval is often the hidden engine that allows AI to ground its outputs in real-world knowledge, find relevant facts, and maintain coherence across conversations or tasks. To truly appreciate how retrieval has become such a critical pillar of modern AI, it is important to first understand how it evolved, from simple keyword matching to sophisticated, learning-driven, and memory-augmented techniques.

在了解现代检索系统之前，简要回顾一下它们的发展历程是有帮助的，如下表所示：

Prior to understanding modern retrieval systems, it is helpful to trace their evolution briefly, which is discussed in the following table:

表 1.1：检索系统的历史时间线

Table 1.1: Historic timelines of retrieval systems

结合表 1.1中给出的背景信息，我们可以清楚地看到，检索不再是简单的查找过程；它已经演变为一个动态的、智能的层，能够主动增强人工智能系统的推理能力。在接下来的章节中，我们将探讨检索系统的工作原理、使其功能强大的关键组件，以及它们如何与生成模型无缝集成，从而构建可靠的、具有上下文感知能力的人工智能应用。

With the preceding background, given in Table 1.1, in mind, it becomes clear that retrieval is no longer a simple lookup process; it has evolved into a dynamic, intelligent layer that actively augments the reasoning capabilities of AI systems. In the following sections, we will explore how retrieval systems work, the key components that make them powerful, and how they integrate seamlessly with generative models to build reliable, context-aware AI applications.

现代检索系统的基础可以追溯到早期的一些创新，例如Facebook AI Research（现为Meta AI）在2020年前后推出的DPR。与传统的稀疏检索方法（例如TF-IDF和BM25）相比，DPR是一项重大突破，因为它为查询和文档都引入了密集向量表示。这使得语义检索成为可能，能够基于语义而非仅仅依赖关键词重叠来查找信息。

The foundation of modern retrieval systems can be traced back to early innovations like DPR, introduced by Facebook AI Research (now Meta AI) around 2020. DPR was a major breakthrough compared to traditional sparse retrieval methods (such as TF-IDF and BM25) because it introduced dense vector representations for both queries and documents. This allowed semantic retrieval, finding information based on meaning rather than relying purely on keyword overlap.

密集检索标志着一个重要的转折点：模型现在可以将查询和文档的含义编码到一个共享的嵌入空间中，从而可以高效地计算相似度。密集检索不再匹配精确的词语，而是匹配概念和思想。然而，早期的密集检索器仍然存在局限性：由于语义匹配粗糙，它们有时会检索到不相关的段落；而且，要将它们扩展到数百万甚至数十亿份文档，就需要解决效率和延迟方面棘手的工程难题。

Dense retrieval marked a major turning point: models could now encode the meaning of a query and a document into a shared embedding space where similarity could be computed efficiently. Instead of matching exact words, dense retrieval matched concepts and ideas. However, early dense retrievers still had limitations: they sometimes retrieved irrelevant passages due to coarse semantic matching, and scaling them to millions or billions of documents required solving difficult engineering challenges around efficiency and latency.

稀疏检索

Sparse retrieval

稀疏检索方法，例如 TF-IDF 和 BM25，依赖于精确匹配关键词和词频统计。虽然这些方法出现较早，但在精确度至关重要且查询与特定术语紧密相关的场景中，例如法律文件检索、科学文献检索和企业文档检索，它们仍然非常有效，因为在这些场景中，精确匹配比一般的语义相似性更为重要。稀疏检索方法与传统的倒排索引技术结合使用也非常高效，并且在许多实际的搜索系统中仍然是强有力的基础。

Sparse retrieval methods like TF-IDF and BM25 rely on matching exact keywords and term frequency statistics. While older, they remain highly effective in cases where precision is critical and queries are closely tied to specific terminology, such as in legal document search, scientific literature, and enterprise document retrieval, where exact matches matter more than general semantic similarity. Sparse retrieval also scales very efficiently with traditional inverted index techniques and remains a strong baseline in many real-world search systems.

密集检索

Dense retrieval

密集检索方法，例如 DPR 和近似最近邻负对比学习密集文本检索( ANCE )模型的引入，标志着检索方式从稀疏词项匹配技术（例如 BM25）向基于语义向量的检索方式的重大转变。密集检索器在处理开放域搜索、歧义查询或同义词和释义常见的场景时表现出色，例如在客户支持机器人、多语言检索或语义常见问题解答( FAQ ) 匹配中。密集检索使系统能够理解问题背后的意图，即使查询和文档中的确切词语有所不同。下图展示了使用向量数据库的语义检索的基本流程：

Dense retrieval methods, introduced with models like DPR and Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (ANCE), marked a major shift from sparse term-matching techniques (e.g., BM25) toward semantic vector-based retrieval. Dense retrievers excel when dealing with open-domain search, ambiguous queries, or when synonyms and paraphrases are common, for example, in customer support bots, multilingual retrieval, or semantic frequently asked questions (FAQs) matching. Dense retrieval allows systems to understand the intent behind a question, even when the exact words differ between the query and the document. The following figure shows the basic flow of semantic retrieval using a vector database:

下图展示了 RAG 流水线的离线阶段。在该阶段，原始文档首先使用语言分块工具（例如基于 Llama 的解析器或 LangChain 工具）进行处理，将其分割成易于管理的片段。然后，这些片段通过嵌入模型（例如 OpenAI 的嵌入 API）生成密集向量表示。生成的嵌入存储在向量数据库中，形成可搜索的索引，为实时查询执行期间的下游检索提供支持。此预处理步骤对于在多模态或基于语言学习模型 (LLM) 的应用中实现快速、可扩展且语义丰富的文档检索至关重要。

The following figure illustrates the offline phase of a RAG pipeline, where raw documents are first processed using language chunking tools (e.g., Llama-based parsers or LangChain utilities) to divide them into manageable segments. These chunks are then passed through an embedding model, such as OpenAI’s embedding API, to generate dense vector representations. The resulting embeddings are stored in a vector database, forming the searchable index that powers downstream retrieval during real-time query execution. This preprocessing step is critical to enabling fast, scalable, and semantically rich document retrieval in multimodal or LLM-based applications.

回顾发展历程，如今的检索系统已经远远超越了早期的DPR架构：

Reflecting on the evolution, today’s retrieval systems have dramatically advanced beyond the early DPR architecture:

• 混合检索：现代系统越来越多地将稀疏检索和密集检索结合起来（例如，BM25 + 密集嵌入），以平衡召回率和精确率，这在长尾查询或特定领域的知识库中尤其有价值。
• Hybrid retrieval: Modern systems increasingly combine sparse and dense retrieval (e.g., BM25 + dense embeddings) to balance recall and precision, especially valuable in long-tail queries or domain-specific knowledge bases.
• 多向量表示：像ColBERT （后期交互模型）这样的高级方法对每个文档编码多个向量而不是单个向量，从而在不牺牲太多速度的情况下提高检索准确率。
• Multi-vector representations: Advanced methods like ColBERT (late interaction models) encode multiple vectors per document rather than a single one, improving retrieval accuracy without sacrificing too much speed.
• 检索器-生成器融合（RAG系统）：检索不再是一个独立的步骤，而是紧密集成到生成流程中。像RAG这样的模型在推理过程中动态检索文档，并对生成的输出进行条件化处理，从而提高事实准确性并减少错误信息。
• Retriever-generator fusion (RAG systems): Retrieval is no longer a standalone step; it is now tightly integrated into the generation pipeline. Models like RAG retrieve documents dynamically during inference and condition the generated output, improving factual accuracy and reducing hallucinations.
• 记忆增强检索：智能人工智能系统利用情景记忆，将外部文档检索与内部学习的知识相结合，从而随着时间的推移不断适应和改进。
• Memory-augmented retrieval: Agentic AI systems use episodic memory, blending external document retrieval with internally learned knowledge to continuously adapt and improve over time.
• 学习检索 (LTR) 和上下文检索：一些较新的架构，如Retro和RePlug，超越了静态索引，使模型本身能够在推理过程中学习检索策略，并根据推理上下文动态地决定要检索的内容。
• Learning-to-retrieve (LTR) and in-context retrieval: Some newer architectures like Retro and RePlug move beyond static indexes, enabling the model itself to learn retrieval strategies during inference, deciding what to retrieve based on the reasoning context dynamically.

此外，向量数据库技术发展迅速。Facebook AI Similarity Search ( Faiss )、Milvus、Qdrant、Azure AI Search 和 Pinecone 等工具提供可扩展的高速向量搜索，支持数十亿个嵌入，并具备近似最近邻( ANN ) 算法、元数据过滤和混合检索功能——所有这些对于驱动现代企业级 RAG 系统都至关重要。

Additionally, vector database technology has matured rapidly. Tools like Facebook AI Similarity Search (Faiss), Milvus, Qdrant, Azure AI Search, and Pinecone offer scalable, high-speed vector search, supporting billions of embeddings with approximate nearest neighbor (ANN) algorithms, metadata filtering, and hybrid retrieval capabilities—all critical for powering modern enterprise-grade RAG systems.

必须认识到，如今的检索不再仅仅是获取文档。它已发展成为一种智能增强机制，涉及过滤、重排序、推理和动态知识基础构建。检索正从后端查找服务演变为下一代人工智能的前端推理组件。

It is crucial to recognize that retrieval today is no longer just about fetching documents. It has become an intelligent augmentation mechanism, involving filtering, reranking, reasoning, and dynamic knowledge grounding. Retrieval is evolving from a backend lookup service into a frontline reasoning component of next-generation AI.

因此，深入理解检索，不仅将其视为一种搜索技术，而且将其视为一种智能增强策略，对于构建可靠、可扩展和目标驱动的新一代 GenAI 应用至关重要。

Thus, understanding retrieval deeply, not simply as a search technique but as an intelligent augmentation strategy, is essential for building reliable, scalable, and goal-driven new-age GenAI applications.

检索系统通常基于召回率@k、精确率@k 和平均倒数排名( MRR ) 等指标进行评估，这些指标衡量系统在搜索结果前列中检索相关文档的效率。我们将在后续章节中更详细地介绍检索评估，但现在需要记住的是，检索质量取决于准确率和排序效率。

Retrieval systems are typically evaluated based on metrics like recall@k, precision@k, and Mean Reciprocal Rank (MRR), which measure how effectively the system retrieves relevant documents among the top results. We will cover retrieval evaluation in greater detail later, but for now, it is important to remember that retrieval quality is judged by both accuracy and ranking efficiency.

发电系统

Generation system

正如我们所见，检索系统侧重于查找最相关的现有信息。然而，许多现实世界的任务需要的不仅仅是检索——它们还需要创建、推理和原创性综合。这正是生成系统发挥作用的地方。

As we have seen, retrieval systems focus on finding the most relevant existing information. However, many real-world tasks demand more than just retrieval—they require creation, reasoning, and original synthesis. This is where generation systems come into play.

在本节中，我们将探讨生成系统的定义、运行机制以及其核心技术。我们将讨论不同类型的生成任务，例如文本、图像和音频的生成，并理解自回归建模、扩散模型和采样策略等关键机制。此外，我们还将介绍温度控制、提示设计以及创造性和事实性之间的平衡等重要概念。

In this section, we will explore what generation systems are, how they operate, and the core techniques that power them. We will discuss different types of generation tasks, such as text, image, and audio creation, and understand key mechanisms like autoregressive modeling, diffusion models, and sampling strategies. Additionally, we will cover important concepts like temperature control, prompt design, and the balance between creativity and factuality.

我们还将探讨生成系统面临的典型挑战，例如幻觉、连贯性问题和安全风险，并重点介绍这些系统真正擅长的领域，尤其是在需要开放式创造力或复杂问题解决能力的任务中。最后，我们将简要介绍现代人工智能架构中如何日益融合检索和生成功能，以构建更贴近现实、更智能的系统。

We will also examine the typical challenges faced by generation systems, such as hallucination, coherence issues, and safety risks, and highlight where these systems truly excel, especially in tasks that demand open-ended creativity or complex problem-solving. Finally, we will briefly introduce how retrieval and generation are increasingly being combined in modern AI architectures to build more grounded and intelligent systems.

让我们首先了解生成系统的基本性质，以及它们与纯粹基于检索的方法有何不同。

Let us begin by understanding the fundamental nature of generation systems and how they differ from purely retrieval-based approaches.

生成系统是一种人工智能模型，其设计目的是生成新内容，而不仅仅是检索现有内容。它们可以通过学习训练数据中的复杂模式来生成文本、图像、音频、代码，甚至是多模态输出。与检索（仅呈现已存在的信息）不同，生成系统使模型能够在推理时动态地编写新句子、创造新图像并解决新问题。

Generation systems are AI models designed to produce new content, rather than simply retrieve it. They can generate text, images, audio, code, and even multimodal outputs by learning complex patterns from training data. Unlike retrieval, which surfaces information that already exists, generation enables models to compose new sentences, invent new images, and solve new problems dynamically at inference time.

现代生成系统通常是大规模神经网络或逻辑层模型（LLM），它们使用跨多个领域的海量数据集，并经过数十亿个参数的训练。下图展示了逻辑层模型和生成模型的类型：

Modern generation systems are typically large-scale neural networks or LLMs trained with billions of parameters on massive datasets across multiple domains. The following figure shows the types of LLMs and generation models:

发电系统类型

Types of generation systems

GenAI系统涵盖多种模态，每种模态都旨在根据用户输入创建文本、图像或音频等内容，展现了现代机器学习（ML ）模型的多功能性和强大功能。让我们来看看生成系统的类型：

GenAI systems span multiple modalities, each designed to create content such as text, images, or audio based on user input, showcasing the versatility and power of modern machine learning (ML) models. Let us look at the types of generation systems:

• 文本生成：像 GPT、Llama 和 Claude 这样的模型擅长生成连贯的段落、回答问题、总结文章、翻译语言，甚至创作诗歌和代码。它们是自回归的，这意味着它们会根据之前的词元预测下一个词元——这使得它们能够逐字构建长而有意义的序列。
• Text generation: Models like GPT, Llama, and Claude specialize in generating coherent paragraphs, answering questions, summarizing articles, translating languages, or even writing poetry and code. They are autoregressive, meaning they predict the next token based on previous tokens—enabling them to build long, meaningful sequences word by word.
• 图像生成：DALL·E、稳定扩散和 Imagen 等模型可根据文本提示生成图像（文本到图像生成）。这些系统依赖于扩散模型或生成对抗网络 (GAN) 等技术，根据用户指令，迭代地从随机噪声中创建逼真的图像。
• Image generation: Models like DALL·E, Stable Diffusion, and Imagen generate images from text prompts (text-to-image generation). These systems rely on techniques like diffusion models or GANs to iteratively create realistic images from random noise, conditioned on user instructions.
• 音频生成：在音频生成领域，像 Whisper（用于自动语音识别）和 VALL-E（用于语音合成）这样的模型可以生成类似人类的语音，甚至可以创作音乐。这些模型学习声波的表征，并根据文本输入识别语音（自动语音识别）或生成音频。
• Audio generation: In audio generation, models like Whisper (for ASR) and VALL-E (for speech synthesis) produce human-like speech or even create music. These models learn representations of sound waves and either recognize speech (ASR) or generate audio based on text inputs.

该生成过程背后的核心技术如下：

Core techniques behind the generation are as follows:

• 语言模型：语言模型经过训练，可以根据先前的序列预测下一个词（标记），因此它们被称为自回归模型，如图 1.3所示。像 GPT-3/4/o3、Llama 或 Claude 这样的大型模型通过自监督学习来学习上下文关系和世界知识，从而能够完成各种生成任务，例如回答问题、总结文档和创意写作。
• Language models: Language models are trained to predict the next word (token) given a previous sequence, and so they are called autoregressive models, as explained in Figure 1.3. Large models like GPT-3/4/o3, Llama, or Claude learn contextual relationships and world knowledge through self-supervised learning, enabling diverse generation tasks such as answering questions, summarizing documents, and creative writing.
• 视觉模型：诸如 DALL·E 和稳定扩散之类的模型将类似 Transformer 的架构应用于图像块或潜在表示，从而实现文本到图像的生成。它们捕捉潜在空间中视觉元素的结构、风格和内容。
• Vision models: Models like DALL·E and Stable Diffusion apply transformer-like architectures to image patches or latent representations, allowing text-to-image generation. They capture the structure, style, and content of visual elements in latent spaces.
• 扩散模型：扩散模型从随机噪声开始，通过迭代去噪来生成逼真的样本。它们常用于生成高保真图像（例如，稳定扩散模型和 Imagen 模型），也被应用于音频甚至 3D 模型生成。扩散模型正被积极应用于语言任务，但与基于 Transformer 的模型（例如 GPT）相比，它们仍不够成熟，也未占据主导地位。语言扩散模型领域正在迅速发展，多项研究表明，基于扩散的生成模型可以与自回归语言模型相媲美，甚至形成互补。
• Diffusion models: Diffusion models start with random noise and iteratively denoise it to create a realistic sample. Popular for generating high-fidelity images (e.g., Stable Diffusion, Imagen), they have also been adapted for audio and even 3D model generation. Diffusion models are being actively adapted for language tasks, though they are still less mature and less dominant than transformer-based models (like GPT). The field of language diffusion models is rapidly evolving, and several research efforts have shown that diffusion-based generative models can be competitive with or complementary to autoregressive language models.

自回归生成

Autoregressive generation

在自回归模型（例如 GPT）中，每个输出标记都是逐个生成的，并以先前生成的标记为条件。这种逐个标记的顺序生成方式使得模型能够产生高度一致的输出，但如果管理不当，也可能导致误差累积。下图解释了 LLM 如何以自回归的方式（一次生成一个标记）进行生成：

In autoregressive models (like GPT), each output token is generated one at a time, conditioned on previously generated tokens. This sequential token-by-token generation allows models to produce highly coherent outputs, but can also lead to error accumulation if not managed carefully. The following figure explains how LLM generates in an autoregressive manner (one token at a time):

以下是温度和采样策略：

The following are the temperature and sampling strategies:

• 温度：控制生成的随机性。温度越低，输出结果越确定、越客观；温度越高，输出结果越具创造性、越多样化。
• Temperature: Controls the randomness of the generation. Lower temperature | more deterministic and factual outputs. Higher temperature | more creative and diverse outputs.
• Top-k 抽样：将下一个标记的选择限制在最有可能的前 k 个标记中。
• Top-k sampling: Limits the next token choice to the top-k most probable tokens.
• Top-p（核心）采样：从累积概率超过 top-p 的最小标记集合中进行选择。
• Top-p (nucleus) sampling: Selects from the smallest set of tokens whose cumulative probability exceeds top-p.

调整这些参数可以对人工智能生成过程中的创造性和精确性进行精细控制。

Tuning these parameters allows fine control over creativity vs. precision in AI generation.

提示策略

Prompting strategies

提示对于引导生成系统的行为至关重要。诸如CoT之类的高级提示技术能够通过鼓励模型在回答问题之前解释其思考过程，从而实现多步骤推理。我们将在下一节中详细解释这些技术。

Prompts are critical for steering the behavior of generation systems. Advanced prompting techniques like CoT enable multi-step reasoning by encouraging models to explain their thought process before answering. We will explain these in more detail in the next section.

了解发电系统的优势所在

Understanding where generation systems excel

发电系统在以下方面尤其强大：

Generation systems are particularly powerful in the following:

• 开放式创意任务（讲故事、创作图像、写诗、编程）。
• Open-ended creativity tasks (storytelling, image creation, poetry, coding).
• 超越检索能力的复杂推理和问题解决能力。
• Complex reasoning and problem-solving beyond retrieval capabilities.
• 个性化和动态响应生成（聊天机器人、教育辅导员）。
• Personalization and dynamic response generation (chatbots, educational tutors).
• 弥补现有数据无法完全匹配查询条件的空白。
• Bridging gaps where no pre-existing data exactly fits the query.

结合检索和生成

Combining retrieval and generation

尽管内容生成系统在创造新内容方面功能强大，但它们有时难以保证事实准确性、知识更新及时，也无法将输出结果与现实世界的信息联系起来。为了克服这些挑战，现代人工智能架构越来越多地将检索和生成功能的优势结合起来，从而催生了一种被称为 RAG 的强大范式。

While generation systems are incredibly powerful at creating new content, they sometimes struggle with factual accuracy, up-to-date knowledge, and grounding their outputs in real-world information. To overcome these challenges, modern AI architectures increasingly combine the strengths of retrieval and generation, giving rise to a powerful paradigm known as RAG.

在下一节中，我们将探讨 RAG 系统如何工作，为什么它们对于构建可靠的 AI 应用程序至关重要，以及它们如何将检索和生成无缝集成到统一的智能工作流程中。

In the next section, we will explore how RAG systems work, why they are critical for building reliable AI applications, and how they seamlessly integrate retrieval and generation into a unified, intelligent workflow.

检索增强生成

Retrieval-augmented generation

RAG 是一种先进的人工智能架构，它将检索和生成整合到一个统一的工作流程中。RAG 系统并非仅仅依赖模型内部的知识（这些知识可能过时或不完整），而是首先检索相关的外部信息，然后根据检索到的内容生成答案。

RAG is an advanced AI architecture that combines retrieval and generation into a unified workflow. Instead of relying solely on a model's internal knowledge (which may be outdated or incomplete), a RAG system first retrieves relevant external information and then generates an answer conditioned on that retrieved content.

RAG 的出现是为了解决纯发电模式面临的主要挑战，这些挑战如下：

RAG emerged to address key challenges faced by pure generation models, which are as follows:

• 幻觉：它有时会产生捏造的、听起来合情合理但实际上不正确的输出。
• Hallucination: It sometimes generates fabricated, plausible-sounding but incorrect outputs.
• 过时的知识：预训练模型的知识库是静态的（截止日期）。
• Stale knowledge: Pre-trained models have a static knowledge base (cutoff dates).
• 真实性：用户通常要求输出结果与可验证的、现实世界的信息相关联。
• Groundedness: Users often demand outputs linked to verifiable, real-world information.

RAG 弥合了这些差距，使输出结果更加准确、更贴近实际、更及时。

RAG bridges these gaps, making outputs more accurate, grounded, and up-to-date.

RAG 工作

RAG working

RAG系统通常包括以下两个主要步骤：

A RAG system typically involves two major steps, which are as follows:

1. 检索步骤：给定用户查询，系统首先从外部知识库（例如，向量数据库）检索前 k 个最相关的文档或数据块。
2. Retrieval step: Given a user query, the system first retrieves the top-k most relevant documents or chunks from an external knowledge base (e.g., a vector database).
3. 生成步骤：将检索到的文档作为上下文传递给语言模型（LLM），LLM 根据检索到的信息生成最终答案。
4. Generation step: The retrieved documents are passed as context to a language model (LLM), which generates the final answer conditioned on the retrieved information.

因此，该模型并非仅由记忆生成；它是先读取，然后推理。

Thus, the model is not generated from memory alone; it is reading first, then reasoning.

基本 RAG 流水线的架构

Architecture of a basic RAG pipeline

以下列表概述了基本的 RAG 流水线结构：

The following list outlines how a basic RAG pipeline looks like:

• 查询理解：对输入的查询进行处理，可以选择重新措辞或扩展，以优化检索。
• Query understanding: The input query is processed, optionally rephrased or expanded, to optimize retrieval.
• 检索：密集型或混合型检索器从向量数据库或搜索引擎中检索最相关的文档。
• Retrieval: A dense or hybrid retriever fetches the most relevant documents from a vector database or search engine.
• 上下文准备：对检索到的文档进行选择、截断、分块和格式化，使其适合 LLM 的输入上下文窗口。
• Context preparation: Retrieved documents are selected, truncated, chunked, and formatted to fit within the LLM’s input context window.
• 生成：LLM 会收到原始查询和检索到的文档，从而生成一个有理有据、上下文丰富的响应。
• Generation: The LLM is prompted with both the original query and the retrieved documents to generate a grounded, contextually rich response.
• 输出交付：将模型的最终响应返回给用户。
• Output delivery: The model's final response is returned to the user.

RAG架构的类型

Types of RAG architectures

如今，根据检索和生成过程的协调方式，出现了许多不同类型的 RAG 架构。但是，为了保持讨论范围的集中性，以下介绍两种最常见且实用的架构：

There are many different types of RAG architectures evolving today, depending on how retrieval and generation are orchestrated. However, to keep the scope focused, the following are the two most common and practical ones:

• 单级 RAG ：
  • 一个简单的流程：检索|生成。
  • 适用于检索质量高且延迟要求低的情况。

  下图展示了一种单级 RAG 架构：

• Single-stage RAG:
  • A simple pipeline: retrieve | generate.
  • Used when retrieval quality is high and latency needs to be minimal.

  The following figure shows a single-stage RAG architecture:

• 两阶段 RAG ：
  • 检索|重排序|生成。
  • 初始检索之后，第二个模型（例如交叉编码器）对文档进行重新排序以提高质量，然后再将其传递给生成器。
  • 通过仅将生成过程集中于最相关的文档，减少幻觉。
• Two-stage RAG:
  • Retrieval | reranking | generation.
  • After initial retrieval, a second model (e.g., cross-encoder) reranks documents to improve the quality before passing them to the generator.
  • Reduces hallucination by focusing the generation only on the most relevant documents.

下图展示了一种两阶段 RAG 架构：

The following figure shows a two-stage RAG architecture:

迭代 RAG

Iterative RAG

以下是两种迭代式 RAG 算法：

The following are the two iterative RAG:

• 检索和生成过程跨越多个回合。
• Retrieval and generation happen across multiple turns.
• 如果第一批文档不足，该模型可以动态检索其他文档，逐步完善答案。
• The model can retrieve additional documents dynamically if the first batch is insufficient, refining the answer step-by-step.

向量数据库和 RAG

Vector databases and RAG

向量数据库是高效 RAG 系统的关键基础设施。

Vector databases are critical infrastructure for efficient RAG systems.

• 用途：它们存储文档嵌入，并基于向量相似性实现快速语义搜索。
• Purpose: They store document embeddings and enable fast semantic search based on vector similarity.
• 示例：Faiss (Meta)、Qdrant、Milvus、Pinecone、Weaviate。
• Examples: Faiss (Meta), Qdrant, Milvus, Pinecone, Weaviate.

ANN 算法具有可扩展性，能够快速找到足够接近的向量，而不是精确匹配，从而实现对数百万或数十亿份文档的实时检索。

ANN algorithms are used for scalability, finding close enough vectors quickly rather than exact matches, enabling real-time retrieval over millions or billions of documents.

向量存储还允许元数据过滤（例如，日期、作者）和分片以实现分布式检索，这对于扩展企业 RAG 系统至关重要。

Vector stores also allow metadata filtering (e.g., date, author) and sharding for distributed retrieval, essential for scaling enterprise RAG systems.

RAG 的快速工程

Prompt engineering for RAG

检索到的内容的格式及其输入到 LLM 的方式会对输出质量产生重大影响。

How the retrieved content is formatted and fed into the LLM significantly affects output quality.

关键技术包括以下几点：

Key techniques include the following:

• 分块：将大篇幅文档拆分成小段，以便将多个段落放入提示中。
• Chunking: Breaking large documents into smaller pieces to fit multiple passages into the prompt.
• 窗口化：在文档上滑动一个固定大小的窗口，以捕获关键字周围的局部上下文。
• Windowing: Sliding a fixed-size window over documents to capture local context around keywords.
• 上下文管理：在不超出模型令牌限制的情况下选择最相关的块。
• Context management: Selecting the most relevant chunks without exceeding the model’s token limit.

精心设计的提示确保语言学习者在生成过程中专注于最重要的信息。

Well-constructed prompts ensure the LLM focuses on the most important information during generation.

高级 RAG 技术

Advanced RAG techniques

随着 RAG 系统的不断发展，人们正在开发先进技术来提高检索质量、提升响应准确率并实现更具上下文感知能力的答案生成。以下是一些先进的 RAG 技术：

As RAG systems evolve, advanced techniques are being developed to enhance retrieval quality, improve response accuracy, and enable more context-aware generation. The following are some of the advanced RAG techniques:

• RAG 重排序：
  • 在生成之前，使用重排序器（如交叉编码器）根据细粒度的相关性评分对检索到的文档进行评估和重新排序。
  • 如果优化得当，可以在不显著增加检索时间的情况下提高精度。
• RAG with reranking:
  • Use a reranker (like a cross-encoder) to evaluate and reorder the retrieved documents based on fine-grained relevance scoring before generation.
  • Improves precision without significantly increasing retrieval time if optimized properly.
• 记忆增强型 RAG ：
  • 检索不仅来自静态知识库，还来自情景记忆——存储过去的对话片段或学习经验。
  • 在多轮对话系统中实现动态、个性化和上下文感知的响应。
• Memory-augmented RAG:
  • Retrieval is not only from static knowledge bases but also from episodic memories-storing past conversation snippets or learned experiences.
  • Enables dynamic, personalized, and context-aware responses in multi-turn dialogue systems.
• 多模态 RAG ：
  • 扩展 RAG 功能，使其能够检索文本和图像（或视频、音频）。
  • 例如：在医疗助理岗位上，同时检索 X 光片和病人病历，并将两者输入到 GPT-4V 或 Flamingo 等多模态模型中。
• Multimodal RAG:
  • Extend RAG to retrieve both text and images (or videos, audio).
  • Example: In a medical assistant role, retrieve x-rays and patient notes together, feeding both into a multimodal model like GPT-4V or Flamingo.

RAG的应用

Applications of RAG

RAG系统已在各行各业迅速普及。让我们来了解一下它的应用：

RAG systems have rapidly gained adoption across industries. Let us understand its applications:

• 企业聊天机器人：基于公司知识库的客户服务机器人。
• Enterprise chatbots: Customer service bots grounded in company knowledge bases.
• 文档质量保证系统：回答来自大型语料库（如研究论文、法律文件或技术手册）的查询。
• Document QA systems: Answering queries from large corpora like research papers, legal documents, or technical manuals.
• 知识管理：组织和动态实时访问企业知识。
• Knowledge management: Organizing and dynamically accessing enterprise knowledge in real-time.
• 个性化人工智能助手：根据用户特定的文档、电子邮件、笔记等定制回复。
• Personalized AI assistants: Tailoring responses based on user-specific documents, emails, notes, etc.

在任何情况下，RAG 都能确保 AI 系统产生可靠、可验证和有理有据的输出。

In every case, RAG ensures the AI system produces reliable, verifiable, and grounded outputs.

人工智能系统中的编排

Orchestration in AI systems

随着人工智能系统变得日益复杂，尤其是在红绿灯（RAG）和智能体人工智能系统兴起之后，智能编排的需求变得至关重要。编排指的是如何动态地管理、排序和协调不同的组件，例如检索引擎、语言模型、内存模块和外部工具，以实现特定目标。

As AI systems become increasingly complex, especially with the rise of RAG and agentic AI systems, the need for intelligent orchestration has become critical. Orchestration refers to how different components, such as retrieval engines, language models, memory modules, and external tools, are managed, sequenced, and coordinated dynamically to achieve a specific goal.

与传统的单次调用 LLM 应用不同，RAG 系统和代理系统涉及多步骤推理和动态决策，需要复杂的编排框架。

Unlike traditional single-call LLM applications, RAG systems and agentic systems involve multi-step reasoning and dynamic decision-making, requiring sophisticated orchestration frameworks.

RAG 系统中的编排

Orchestration in RAG systems

在 RAG 系统中，编排涉及以下内容：

In RAG systems, orchestration involves the following:

• 查询理解：在检索之前预处理用户查询。
• Query understanding: Preprocessing user queries before retrieval.
• 文档检索：与矢量数据库（例如 Faiss、Qdrant、Pinecone）对接，获取前 k 个相关文档。
• Document retrieval: Interfacing with vector databases (e.g., Faiss, Qdrant, Pinecone) to fetch top-k relevant documents.
• 上下文准备：对检索到的文档进行分块、选择和格式化，使其适合 LLM 的上下文窗口。
• Context preparation: Chunking, selecting, and formatting retrieved documents to fit within the LLM’s context window.
• 提示构建：将检索到的知识动态地插入到结构良好的提示中。
• Prompt construction: Dynamically inserting retrieved knowledge into well-structured prompts.
• 响应生成：使用 LLM 生成基于所提供文档的输出。
• Response generation: Using the LLM to generate outputs grounded in the provided documents.
• 后处理（可选）：过滤、重新排序或验证模型输出。
• Post-processing (optional): Filtering, reranking, or verifying model outputs.

LangChain、LlamaIndex 和 Haystack 等框架专门用于自动编排这些步骤，从而更容易构建可扩展且可用于生产的 RAG 管道。

Frameworks like LangChain, LlamaIndex, and Haystack specialize in orchestrating these steps automatically, making it easier to build scalable and production-ready RAG pipelines.

下图解释了 LangChain 如何协调整个 RAG 流程：

The following figure explains how LangChain is orchestrating the entire RAG process:

良好的 RAG 流程编排可确保以下几点：

Good RAG orchestration ensures the following:

• 最低延迟
• Minimal latency
• 高检索质量
• High retrieval quality
• 检索和生成之间的紧密耦合
• Tight coupling between retrieval and generation
• 对令牌限制和内存的稳健处理
• Robust handling of token limits and memory

智能体系统中的编排

Orchestration in agentic systems

在智能体系统中，编排变得更加动态。

In agentic systems, orchestration becomes even more dynamic.

智能体是一种人工智能实体，能够执行以下操作：

An agent is an AI entity capable of the following:

• 对任务进行推理。
• Reasoning about a task.
• 选择操作（例如，工具使用、API 调用、检索）。
• Choosing actions (e.g., tool usage, API calls, retrievals).
• 逐步执行操作。
• Executing actions step-by-step.
• 根据中期结果动态地反思和调整其计划。
• Reflecting and adjusting its plan dynamically based on intermediate results.

智能体编排包括以下几个方面：

Agentic orchestration involves the following:

• 工具选择：根据当前目标决定要调用哪些外部工具或函数。
• Tool selection: Deciding which external tools or functions to call based on the current goal.
• 多步骤规划：按逻辑顺序采取行动，以解决复杂问题。
• Multi-step planning: Sequencing actions logically toward solving complex problems.
• 记忆管理：保留过去的操作、中间结果和对话历史记录，以指导未来的步骤。
• Memory management: Retaining past actions, intermediate results, and conversation history to guide future steps.
• 错误处理：如果操作失败，则进行重试或优雅恢复。
• Error handling: Retrying or recovering gracefully if an action fails.
• 目标管理：持续检查最终任务目标是否已实现。
• Goal management: Continuously checking whether the final task objective has been met.

LangChain Agents、LamaIndex、Haystack 等框架为代理系统提供编排原语，使 AI 模型能够像自主的多步骤决策者一样行事。

Frameworks like LangChain Agents, LamaIndex, Haystack, etc., provide orchestration primitives for agentic systems, allowing AI models to behave like autonomous, multi-step decision-makers.

良好的代理编排可确保以下几点：

Good agentic orchestration ensures the following:

• 将任务分解为可管理的步骤。
• Task decomposition into manageable actions.
• 动态适应不可预见的结果。
• Dynamic adaptation to unforeseen outcomes.
• 类似人类的推理链，跨越多个工具交互。
• Human-like reasoning chains that span multiple tool interactions.

编排侧重于管理复杂人工智能系统的整体流程，而另一个基础概念则在更低的层面上运作——即信息本身在模型内部的表示和处理方式。在进行任何检索、生成或推理之前，输入文本必须被分解成模型可以理解的形式——这个过程称为分词。

While orchestration focuses on managing the overall flow of complex AI systems, another foundational concept operates at a much lower level - how information itself is represented and processed inside models. Before any retrieval, generation, or reasoning can happen, input text must be broken down into a form that models can understand—a process known as tokenization.

要充分理解人工智能系统的能力和局限性，就必须了解什么是令牌、令牌化是如何工作的，以及为什么它在塑造性能、成本和设计选择方面起着至关重要的作用。

To fully appreciate the capabilities and limitations of AI systems, it is essential to understand what tokens are, how tokenization works, and why it plays a critical role in shaping performance, cost, and design choices.

现在让我们来了解一下分词。

Let us now understand tokenization.

人工智能系统中的令牌

Tokens in AI systems

在现代人工智能系统，特别是语言学习模型（LLM）中，词元（token）的概念对于输入和输出的处理至关重要。词元不一定是一个词；它可以是一个词、词的一部分（子词），甚至是标点符号和特殊字符，具体取决于模型的词元生成器。

In modern AI systems, particularly LLMs, the concept of tokens is fundamental to how inputs and outputs are processed. A token is not necessarily a word; it can be a word, a part of a word (subword), or even punctuation and special characters, depending on the model’s tokenizer.

分词是将文本分解成模型可以理解和处理的离散单元的过程。像 GPT-3、GPT-4 和 Llama 这样的模型并不直接处理原始文本，而是处理词元序列。

Tokenization is the process of breaking down text into discrete units that the model can understand and process. Models like GPT-3, GPT-4, and Llama do not operate directly on raw text; they operate on sequences of tokens.

代币化策略有多种类型，例如以下几种：

There are different types of tokenization strategies, like the following:

• 词级分词：在早期模型中，一个词元通常对应一个完整的单词（例如，dog ，running ）。这种方法虽然简单，但处理罕见词或复合词时效率低下。
• Word-level tokenization: In early models, one token often corresponded to one full word (e.g., dog, running). This approach is simple but inefficient for handling rare or compound words.
• 子词级分词 [字节对编码(BPE)] ：被大多数现代语言模型（包括 GPT、BERT）采用。它将常用词根合并（例如，run + ning | running ），以减少词汇量，同时仍能高效地处理罕见词。
• Subword-level tokenization [Byte-Pair Encoding (BPE)]: Used by most modern LLMs (including GPT, BERT). Common word parts are merged (e.g., run + ning | running) to reduce the vocabulary size while still handling rare words efficiently.
• 字符级分词：每个字符（包括空格和标点符号）都被视为一个词元。这种方法会显著增加序列长度，并且由于计算成本高昂，在大型模型中不太常用。
• Character-level tokenization: Each character (including spaces and punctuation) is treated as a token. It increases sequence length dramatically and is less common in large models due to computational cost.
• 字节级分词：有些模型基于原始字节进行分词（例如，GPT-2 使用字节级 BPE），无需复杂的预分词即可实现灵活的多语言处理。
• Byte-level tokenization: Some models tokenize based on raw bytes (e.g., GPT-2 uses byte-level BPE), allowing flexible multilingual processing without complex pre-tokenization.

代币数量决定：

The number of tokens determines:

• 费用（例如 OpenAI 的 GPT-4 等 API 的定价是按 1,000 个代币计算的）。
• The cost (for APIs like OpenAI's GPT-4 pricing is per 1,000 tokens).
• 上下文窗口大小（模型一次可以看到多少信息）。
• The context window size (how much information the model can see at once).
• 如果输入超过最大标记限制（例如，GPT-4 turbo 最大支持 128K 个标记），则会发生模型行为和截断。
• Model behavior and the truncation happen if input exceeds maximum token limits (e.g., GPT-4 turbo supports 128K tokens max).

因此，了解令牌、令牌代表什么以及如何计数对于优化性能、控制生成长度、管理成本和设计有效的提示工程策略至关重要。

Thus, understanding tokens, what they represent, and how they are counted is critical for optimizing performance, controlling generation length, managing costs, and designing effective prompt engineering strategies.

例如，输入“Today is a beautiful day outside .”可能会被拆分成子词，如（To, day, is, a, be, aut, iful , day , out , side），具体取决于分词器。

For example, the input Today is a beautiful day outside. might be split into subwords like (To, day, is, a, be, aut, iful, day, out, side) depending on the tokenizer.

词元被分割成词元后，每个词元都会使用词汇表（在模型训练期间预先构建）映射到唯一的词元 ID。每个词元 ID 都对应一个模型内部可以理解的整数。例如：

Once split into tokens, each token is then mapped to a unique token ID using a vocabulary table (pre-built during model training). Each token ID corresponds to an integer that the model understands internally. For instance:

• 至| 98
• To | 98
• 第1452天
• day | 1452
• 美丽（拆分成子词）| 2932 和 1709
• beautiful (split into subwords) | 2932 and 1709

因此，整个输入序列被转换为标记 ID 向量——模型可以对其进行操作的数字列表。

Thus, the entire input sequence is transformed into a vector of token IDs—a list of numbers that the model can operate on.

此时，词元 ID 会经过一个嵌入层。该嵌入层将每个词元 ID 转换为一个高维向量（例如，768 维向量），该向量能够捕捉词元之间的语义关系。语义相关的词元（例如，“狗”和“小狗”）在向量空间中的嵌入向量会非常接近。

At this point, the token IDs are passed through an embedding layer. This embedding layer converts each token ID into a high-dimensional vector (e.g., 768-dimensional) that captures semantic relationships between tokens. Tokens that are semantically related (e.g., dog and puppy) will have embeddings that are close in vector space.

从那里开始，词元嵌入会流经模型的架构、注意力层、Transformer 模块，最终生成输出或进行推理。

From there, the token embeddings move through the model’s architecture, the attention layers, transformer blocks, and eventually lead to output generation or reasoning.

总而言之，分词弥合了人类语言和机器理解之间的鸿沟。它将杂乱无章、长度不一的人类文本转换为标准化的数字。深度学习模型可以高效处理的形式。如果没有分词，现代语言模型将无法处理人类交流的复杂性和多样性。下图展示了现代语言模型中的分词流程：

In summary, tokenization bridges the gap between human language and machine understanding. It translates messy, variable-length human text into standardized numerical forms that deep learning models can efficiently process. Without tokenization, modern language models would not be able to handle the complexity and diversity of human communication. The following figure shows the tokenization process flow in modern language models:

向量数据库

Vector database

虽然分词使语言模型能够以细粒度的方式处理和理解文本输入，但处理大规模检索任务需要另一种表示方法。检索系统不再直接处理词元，而是基于密集向量嵌入——一种能够捕捉文本、图像或其他数据类型语义的数学表示。为了高效地存储、搜索和检索这些嵌入，向量数据库已成为现代人工智能架构的重要组成部分。

While tokenization enables language models to process and understand text inputs at a granular level, handling large-scale retrieval tasks requires a different kind of representation. Instead of working directly with tokens, retrieval systems operate on dense vector embeddings—mathematical representations that capture the semantic meaning of text, images, or other data types. To store, search, and retrieve these embeddings efficiently, vector databases have become an essential component of modern AI architectures.

现在让我们来探讨向量数据库的作用，以及它们如何为可扩展的高性能检索系统提供支持。

Let us now explore the role of vector databases and how they power scalable, high-performance retrieval systems.

在进一步探讨向量数据库之前，首先需要了解它们在其他类型数据库中的定位。

Before we explore vector databases further, it is important to first understand where they fit among other types of databases.

让我们来看一下数据库的类型，它们如下：

Let us look at the types of databases, which are as follows:

• 关系型数据库：将数据组织成由行和列组成的结构化表格；最适合使用 SQL 查询的结构化数据。（例如，MySQL、PostgreSQL）
• Relational databases: Organize data into structured tables of rows and columns; best for structured data with SQL querying. (e.g., MySQL, PostgreSQL)
• 键值存储：存储简单的键值对；针对快速查找进行了优化。（例如，Redis、DynamoDB）。
• Key-value stores: Store simple key-value pairs; optimized for fast lookups. (e.g., Redis, DynamoDB).
• 文档数据库：以灵活的格式（例如JavaScript 对象表示法( JSON )）管理半结构化数据。（例如，MongoDB、CouchDB）。
• Document databases: Manage semi-structured data in flexible formats like JavaScript Object Notation (JSON). (e.g., MongoDB, CouchDB).
• 图数据库：使用节点和边存储实体之间的关系。（例如，Neo4j、ArangoDB）。
• Graph databases: Store relationships between entities using nodes and edges. (e.g., Neo4j, ArangoDB).
• 宽列数据库：将数据组织成列族，以便高效地进行大规模读/写操作。（例如，Cassandra、HBase）。
• Wide-column databases: Organize data into column families for efficient large-scale reads/writes. (e.g., Cassandra, HBase).
• 内存数据库：将数据存储在随机存取存储器（RAM ）中，以实现超快访问。（例如，Redis、Memcached）。
• In-memory databases: Store data in random access memory (RAM) for ultra-fast access. (e.g., Redis, Memcached).
• 时间序列数据库：专门用于存储顺序的、带时间戳的数据。（例如，InfluxDB、TimescaleDB）。
• Time-series databases: Specialized for sequential, time-stamped data. (e.g., InfluxDB, TimescaleDB).
• 文本搜索数据库：针对全文索引和搜索进行了优化。（例如，Elasticsearch、Solr）。
• Text search databases: Optimized for full-text indexing and search. (e.g., Elasticsearch, Solr).
• 空间数据库：存储地理/空间数据。（例如，PostGIS）。
• Spatial databases: Store geographic/spatial data. (e.g., PostGIS).
• Blob 存储：管理大型二进制文件，例如图像和视频。[例如，Amazon Simple Storage Service ( S3 )]。
• Blob stores: Manage large binary files like images and videos. [e.g., Amazon Simple Storage Service (S3)].
• 账本数据库：不可变记录保存[例如，Hyperledger Fabric、Amazon Quantum Ledger Database ( QLDB )]。
• Ledger databases: Immutable record-keeping [e.g., Hyperledger Fabric, Amazon Quantum Ledger Database (QLDB)].
• 层次数据库：树状父子结构数据。[例如，IBM信息管理系统( IMS )] 。
• Hierarchical databases: Tree-like parent-child structured data. [e.g., IBM Information Management System (IMS).
• 向量数据库：存储用于语义搜索和相似性操作的高维嵌入。（例如，Faiss、Chroma、Pinecone）。
• Vector databases: Store high-dimensional embeddings for semantic search and similarity operations. (e.g., Faiss, Chroma, Pinecone).

其中，向量数据库已成为人工智能、检索和智能推理系统不可或缺的一部分。

Among these, vector databases have become essential for AI, retrieval, and agentic reasoning systems.

了解向量数据库

Understanding vector databases

向量数据库旨在存储和检索稠密向量嵌入，即文本、图像或音频等非结构化数据的数值表示。与关系型数据库或文档型数据库不同，向量数据库基于余弦相似度或欧氏距离等距离度量进行相似性搜索，而不是进行精确匹配。

Vector databases are designed to store and retrieve dense vector embeddings, numerical representations of unstructured data such as text, images, or audio. Unlike relational or document databases, vector databases perform similarity searches based on distance metrics like cosine similarity or Euclidean distance rather than exact matching.

它们使 AI 模型能够高效地检索语义相似的项目，这是 RAG 和内存增强型代理系统的关键操作。

They enable AI models to retrieve semantically similar items efficiently, a critical operation for RAG and memory-augmented agentic systems.

向量数据库中的索引算法

Indexing algorithms in vector databases

高效的向量搜索是人工智能驱动应用的基础，但对数百万个高维向量进行穷举搜索计算量巨大。为了解决这一问题，我们需要……为了应对这一挑战，向量数据库使用专门的索引算法来提高搜索速度，同时兼顾准确性和内存效率。

Efficient vector search is fundamental to AI-driven applications, but performing brute-force searches over millions of high-dimensional vectors is computationally intensive. To address this challenge, vector databases use specialized indexing algorithms that improve search speed while balancing accuracy and memory efficiency.

以下是一些常用的索引技术：

The following are some commonly used indexing techniques:

• 扁平搜索（暴力搜索）：每个查询都会与所有存储的向量进行比较。
  • 提供最高精度。
  • 速度慢，且无法扩展到大型数据集。
• Flat (brute-force search): Every query is compared against all stored vectors.
  • Offers the highest accuracy.
  • Slow and not scalable for large datasets.
• 倒排文件索引（IVF）：向量被分组到簇中，搜索仅限于最相关的簇。
  • 比蛮力方法快得多。
  • 在保持良好精度的同时，提高了效率。
• Inverted File Indexing (IVF): Vectors are grouped into clusters, and searches are restricted to the most relevant clusters.
  • Significantly faster than brute-force.
  • Maintains good accuracy with improved efficiency.
• 分层可导航小世界（HNSW）：一种基于图的结构，能够实现快速准确的ANN搜索。
  • 非常适合实时搜索。
  • 兼顾高记忆率和低延迟。
• Hierarchical navigable small world (HNSW): A graph-based structure that enables fast and accurate ANN searches.
  • Well-suited for real-time search.
  • Balances high recall with low-latency.
• 乘积量化（PQ）：将向量压缩成紧凑的表示形式，以实现可扩展的搜索。
  • 适用于超大型数据集。
  • 存储和检索效率高，准确度也相当不错。
• Product quantization (PQ): Compresses vectors into compact representations for scalable search.
  • Suitable for very large datasets.
  • Efficient in both storage and retrieval with reasonable accuracy.

每种索引方法在速度、准确性和内存使用方面都各有侧重。最佳选择取决于应用程序的具体需求，包括数据集大小、性能要求和基础设施限制。

Each of these indexing methods offers a different balance between speed, accuracy, and memory usage. The best choice depends on the specific needs of the application, including dataset size, performance requirements, and infrastructure constraints.

向量数据库中的搜索算法

Search algorithms in vector databases

向量数据库旨在高效地存储和检索高维向量嵌入，这对于驱动现代人工智能应用（例如语义搜索、推荐系统和图像相似度分析）至关重要。数据被编码成向量嵌入并建立索引后，即可使用搜索算法检索给定查询向量的最近邻向量。

Vector databases are designed to store and retrieve high-dimensional vector embeddings efficiently, key to powering modern AI applications like semantic search, recommendation systems, and image similarity. Once data is encoded into vector embeddings and indexed, search algorithms are used to retrieve the nearest neighbors to a given query vector.

两种主要方法如下：

The two main approaches are as follows:

• 精确搜索：这种方法将查询向量与数据库中的每个向量进行比较，以找到最相似的向量。它具有较高的召回率和准确率，但计算量大且速度慢，因此仅适用于小型数据集或离线分析。
• Exact search: This method compares the query vector against every vector in the database to find the most similar ones. It offers high recall and accuracy but is computationally expensive and slow, making it suitable only for small datasets or offline analysis.
• 人工神经网络搜索：人工神经网络算法并非进行穷举比较，而是搜索数据库的较小子集以找到足够好的结果。这种搜索方式在速度和结果之间取得了平衡。精度对于扩展数百万或数十亿个向量至关重要，从而实现实时搜索和推理。
• ANN search: Instead of exhaustive comparison, ANN algorithms search for a smaller subset of the database to find results that are good enough. This trade-off between speed and precision is essential for scaling millions or billions of vectors, enabling real-time search and inference.

常用的ANN技术包括以下几种：

Common ANN techniques include the following:

• HNSW ：一种基于图的方法，以快速准确的检索而闻名。
• HNSW: A graph-based method known for fast and accurate retrieval.
• IVF + PQ ：它结合了聚类和压缩，以提高内存效率。
• IVF + PQ: It combines clustering and compression for memory efficiency.
• ScaNN ：谷歌开发的一种高性能 ANN 算法，针对大规模生产环境进行了优化。
• ScaNN: A high-performance ANN algorithm developed by Google, optimized for large-scale production environments.

大多数生产级向量数据库（如 Faiss、Milvus 或 Pinecone）都依赖 ANN 搜索来实现低延迟、高吞吐量的性能，而不会在相关性或召回率方面做出太大牺牲。

Most production-grade vector databases (like Faiss, Milvus, or Pinecone) rely on ANN search to deliver low-latency, high-throughput performance without sacrificing too much on relevance or recall.

嵌入和嵌入模型

Embeddings and embedding models

向量数据库的核心是嵌入的概念。嵌入是一个稠密的数值向量，它以一种方式捕捉输入（文本、图像、音频）的语义含义，使得相似的输入在向量空间中彼此更接近。

At the core of vector databases lies the concept of embeddings. Embedding is a dense numerical vector that captures the semantic meaning of an input (text, image, audio) in a way that similar inputs are closer together in the vector space.

例如，即使措辞不同，两个关于狗的句子也会有紧密相邻的嵌入。

For example, two sentences about dogs will have embeddings close together even if they use different wording.

嵌入模型是经过训练的神经网络，可以将输入映射到这些向量空间。一些常见的嵌入模型类型如下：

Embedding models are neural networks trained to map inputs into these vector spaces. Some popular types of embedding models are as follows:

• 文本嵌入：
  • OpenAI 的 text-embedding-ada-002
  • 句子 BERT ( SBERT )
  • 拥抱脸迷你LM
• Text embeddings:
  • OpenAI's text-embedding-ada-002
  • Sentence-BERT (SBERT)
  • Hugging Face MiniLM
• 图像嵌入：
  • CLIP（OpenAI）
  • DNOv2（元）
• Image embeddings:
  • CLIP (OpenAI)
  • DINOv2 (Meta)
• 多模态嵌入：
  • CLIP（联合视觉语言）
  • 火烈鸟（DeepMind）
• Multimodal embeddings:
  • CLIP (joint vision-language)
  • Flamingo (DeepMind)

嵌入模型至关重要，因为检索质量很大程度上取决于嵌入的质量。

Embedding models are crucial because the quality of retrieval depends heavily on the quality of embeddings.

向量数据库对 RAG 和智能体系统的重要性

Importance of vector databases for RAG and agentic systems

在 RAG 管道中，查询的嵌入与向量数据库中存储的文档嵌入进行匹配，以检索最相关的知识来确定响应。

In RAG pipelines, embeddings of queries are matched against stored document embeddings inside a vector database to retrieve the most relevant knowledge for grounding responses.

在智能体系统中，智能体可能需要：

In agentic systems, an agent might need to:

• 回顾过去的经历
• Retrieve past experiences
• 获取外部知识
• Fetch external knowledge
• 搜索工具输出结果
• Search through the tool outputs

所有操作均在运行时动态进行，基于向量相似度，而不仅仅是关键词匹配。

All dynamically at runtime, based on vector-based similarity, not just keyword matching.

因此，向量数据库能够实现语义记忆和可扩展的智能检索，这两点是新一代人工智能的基石。下图展示了使用嵌入的类似搜索流程，其中输入数据被转换为向量以检索语义相似的结果：

Thus, vector databases enable semantic memory and scalable, intelligent retrieval, two cornerstones of the new age of GenAI. The following figure represents the workflow of a similar search pipeline using embeddings, where input data is transformed into vectors to retrieve semantically similar results:

虽然向量数据库能够快速高效地检索语义相似的文档，但相似度搜索返回的结果并不总是与用户的真实意图完全一致。单纯基于向量相似度的检索有时会检索到相关性不高的文档，导致最终结果准确性降低或缺乏依据。

While vector databases enable fast and efficient retrieval of semantically similar documents, the top results returned by similarity search are not always perfectly aligned with the user's true intent. Retrieval-based purely on vector similarity can sometimes surface documents that are only loosely relevant, leading to less accurate or less grounded final outputs.

为了应对这一挑战，通常会引入一个重要的优化步骤，称为重排序。重排序允许人工智能系统基于更深层次的相关性评分对检索到的文档进行重新排序和优先级排序，从而提高最终传递给语言模型进行生成的输入质量。

To address this challenge, an important refinement step called reranking is often introduced. Reranking allows AI systems to reorder and prioritize retrieved documents based on deeper relevance scoring, improving the quality of the inputs ultimately passed to the language model for generation.

现在让我们来了解一下重排序，为什么需要它，它是如何工作的，以及现代人工智能流程中使用的不同方法。

Let us now understand reranking, why it is needed, how it works, and the different approaches used in modern AI pipelines.

重新排名

Reranking

重新排序的概念对人工智能来说并不新鲜，它在推荐系统和搜索引擎中有着深厚的渊源。

The concept of reranking is not new to AI. It has deep roots in recommendation systems and search engines.

在传统的推荐流程（例如，推荐产品、电影、文章）中，系统通常会根据用户偏好检索一大堆候选结果，例如排名前 100 或前 1000 的项目。基于用户历史记录或内容相似度等粗略匹配，检索系统会进行初步筛选。然而，这些初始候选结果往往并不完美，因为检索系统优先考虑召回率，力求尽可能多地检索到潜在的有效条目，即使这会牺牲精确度。

In traditional recommendation pipelines (e.g., recommending products, movies, articles), the system typically retrieves a broad set of candidates, say, the top 100 or top 1000 items, based on rough matching like user history or content similarity. However, these initial candidates are often imperfect, as retrieval systems prioritize recall, getting as many potentially good items as possible, even at the cost of precision.

因此，引入了重新排序步骤，详情如下：

Thus, a reranking step is introduced, details as follows:

• 更复杂的模型（通常是更深层的神经网络）会重新评估初始数据集，并根据更准确的相关性得分重新排序。
• A more sophisticated model (often a deeper neural network) re-evaluates the initial set and reorders them based on more accurate relevance scores.
• 目标是最大限度地提高搜索结果顶部的精准度，确保向用户展示的前几个项目是最相关、最有影响力的。
• The goal is to maximize precision at the top and ensure the first few items shown to the user are the most relevant and impactful.

这种两阶段方法，即检索和重新排序，现在不仅是推荐系统的基础，而且也是现代RAG管道和搜索引擎的基础。

This two-stage approach, retrieval and reranking, is now fundamental not just in recommendation systems but also in modern RAG pipelines and search engines.

双编码器与交叉编码器

Bi-encoders vs. cross-encoders

双编码器和交叉编码器是自然语言处理（NLP ）中用于语义搜索和排序等任务的两种常用架构。

Bi-encoders and cross-encoders are two popular architectures used for tasks like semantic search and ranking in natural language processing (NLP).

双编码器使用同一模型将查询和文档独立编码成不同的向量嵌入。然后可以使用余弦相似度或其他距离度量有效地比较这些嵌入，这使得双编码器非常适合对速度和可扩展性要求极高的大规模检索。

Bi-encoders independently encode the query and document into separate vector embeddings using the same model. These embeddings can then be efficiently compared using cosine similarity or other distance metrics, making bi-encoders ideal for large-scale retrieval where speed and scalability are critical.

另一方面，交叉编码器将查询和文档联合编码，并将它们一起输入到Transformer模型中。这使得模型能够考虑词元之间的交叉注意力，从而获得更准确的相关性评分。然而，这种方法计算量大且速度慢，限制了其在实时或大规模系统中的应用。

Cross-encoders, on the other hand, jointly encode the query and document by feeding them together into a transformer model. This allows the model to consider cross-attention between tokens, resulting in more accurate relevance scoring. However, this approach is computationally expensive and slower, limiting its use in real-time or large-scale systems.

在检索和重排序的背景下，一种常见的模式是使用双向编码器进行快速候选检索，然后使用交叉编码器对排名靠前的结果进行重排序以提高精度，从而有效地平衡效率和准确性：

In the context of retrieval and reranking, a common pattern is to use bi-encoders for fast candidate retrieval, followed by cross-encoders for reranking the top results to improve precision, balancing efficiency and accuracy effectively:

• 双编码器：
  • 将查询和文档分别编码成稠密向量。
  • 相似度是在编码之后计算的，通常通过点积或余弦相似度来计算。
  • 速度极快且可扩展，非常适合从大型语料库中进行首次检索。
  • 例如：DPR、句子转换器。
• Bi-encoders:
  • Encode the query and documents separately into dense vectors.
  • Similarity is computed after encoding, usually via dot product or cosine similarity.
  • Extremely fast and scalable, and ideal for first-pass retrieval from large corpora.
  • Examples: DPR, Sentence Transformers.
• 交叉编码器：
  • 将查询和文档连接起来，并一起进行编码。
  • 该模型将这两个部分结合起来，可以模拟细粒度的交互作用。
  • 比双向编码器慢得多，因为每个查询-文档对都需要重新计算。
  • 然而，它在判断真正的相关性方面要准确得多。
  • 例如：BERT 用于段落排序，MiniLM 用于重排序。
• Cross-encoders:
  • Concatenate the query and document together and encode them jointly.
  • The model sees both pieces together and can model fine-grained interactions.
  • Much slower than bi-encoders because every query-document pair needs to be recomputed.
  • However, much more accurate at judging true relevance.
  • Examples: BERT for passage ranking, MiniLM for reranking.

用于重排序的交叉编码器

Cross-encoders for reranking

在 RAG 流程或 AI 搜索系统中，典型的工作流程如下：

In RAG pipelines or AI search systems, the typical workflow is as follows:

• 使用双编码器检索器快速获取前 k 个候选结果。
• Use a bi-encoder retriever to fetch top-k candidates quickly.
• 对这k 个候选者（例如，前 100 个）应用交叉编码器重排序器，以重新评分并重新排序。
• Apply a cross-encoder reranker on these k candidates (e.g., top 100) to rescore and reorder them.
• 选择排名前 n 的重新排名文档（例如，前 5 个或前 10 个）输入到生成模型中。
• Select the top-n reranked documents (e.g., top 5 or top 10) to feed into the generation model.

由于交叉编码器模型在查询文档和候选文档之间实现了深度交互，因此显著提高了检索信息的质量，从而在生成过程中产生了更有依据、更准确、更具上下文相关性的输出。

Due to the cross-encoder model’s deep interactions between the query and candidate documents, it significantly improves the quality of retrieved information, leading to better-grounded, more accurate, and more contextually relevant outputs during generation.

因此，重排序，特别是使用交叉编码器的重排序，是当今构建高精度、生产级 AI 检索系统的重要工具，如下图所示：

Thus, reranking, especially using cross-encoders, is a vital tool in building high-precision, production-grade AI retrieval systems today, as shown in the following figure:

虽然重排序可以提高检索信息的质量和相关性，但它并不能从根本上保证人工智能的最终输出始终安全、公正或符合应用需求。即使检索精度很高，生成模型仍然可能出现偏差，引入敏感内容，或生成与用户预期不符的输出。为了应对这些风险，现代人工智能系统实施了防护机制、结构化控制和验证机制，旨在监控、过滤和调整模型行为。下一节我们将探讨防护机制的概念、其重要性以及它们在检索和生成流程中的应用。

While reranking improves the quality and relevance of retrieved information, it does not inherently guarantee that the AI’s final output will always be safe, unbiased, or aligned with application requirements. Even with high-precision retrieval, generation models can still hallucinate, introduce sensitive content, or produce outputs that deviate from user expectations. To address these risks, modern AI systems implement guardrails, structured controls, and validation mechanisms designed to monitor, filter, and shape model behavior. In the next section, we will explore the concept of guardrails, why they are essential, and how they are applied across retrieval and generation pipelines.

护栏

Guardrails

随着人工智能系统能力的不断提升，设置防护机制（即引导和约束模型行为的结构化控制措施）变得至关重要。防护机制确保模型即使在处理复杂、开放式的输入时，也能安全、合乎伦理地运行，并符合应用或组织的目标。

As AI systems become increasingly capable, the need for guardrails, structured controls that guide and constrain model behavior, has become critical. Guardrails ensure that models act safely, ethically, and in alignment with application or organizational goals, even when handling complex, open-ended inputs.

虽然重新排序有助于呈现更多相关且真实的信息，但它本身并不能防止幻觉、偏见传播、政策违规或用户操纵。逻辑逻辑模型（LLM）功能强大但缺乏确定性；即使输入数据干净，如果不加以控制，也可能产生不安全、冒犯性或误导性的输出。安全防护措施有助于维护信任、安全性和合规性。在现实世界环境中部署人工智能系统时，尤其是在企业、医疗保健、金融和教育领域，这些都是至关重要的因素。下图展示了一个增强了安全防护措施、重新排序和基于逻辑逻辑模型的响应生成的红绿灯（RAG）系统的架构：

While reranking helps in surfacing more relevant and factual information, it does not inherently prevent hallucinations, bias propagation, policy violations, or user manipulation. LLMs are powerful but non-deterministic; even with clean inputs, they can produce unsafe, offensive, or misleading outputs if left unchecked. Guardrails help maintain trust, safety, and compliance. They are all crucial factors when deploying AI systems in real-world environments, especially in the enterprise, healthcare, finance, and education sectors. The following figure illustrates the architecture of a RAG system enhanced with guardrails, reranking, and LLM-based response generation:

护栏类型

Types of guardrails

护栏通常分两个主要阶段运行，具体如下：

Guardrails typically operate at two major stages, which are described in the following list:

• 输入防护机制：它们会在问题用户查询到达模型之前对其进行过滤、重述或阻止。例如：
  • 检测并阻止恶意提示（例如，要求模型生成有害内容）。
  • 将模糊、模棱两可或有风险的查询改写成更安全的形式。
  • 在提示中添加免责声明或限制条件，以明确对模型行为的预期。
• Input guardrails: They filter, rephrase, or block problematic user queries before they reach the model. Examples include:
  • Detecting and blocking malicious prompts (e.g., asking the model to generate harmful content).
  • Rewriting vague, ambiguous, or risky queries into safer forms.
  • Adding disclaimers or constraints to the prompt to set clear expectations for model behavior.
• 输出防护机制：输出防护机制在将模型生成的输出交付给用户之前，对其进行分析、验证和修改。例如：
  • 删除有害、有偏见或不安全的内容。
  • 确保产出符合组织政策（例如，不共享个人数据）。
  • 在显示答案之前，会先根据可信的知识库进行事实核查。
  • 强制执行语气、风格或内容格式要求（例如，专业、中立）。
• Output guardrails: Output guardrails analyze, validate, and modify model-generated outputs before they are delivered to the user. Examples include:
  • Removing toxic, biased, or unsafe content.
  • Ensuring outputs comply with organizational policies (e.g., no sharing of personal data).
  • Fact-checking answers against trusted knowledge bases before displaying them.
  • Enforcing tone, style, or content formatting requirements (e.g., professional, neutral).

护栏的安装方法

Methods of applying guardrails

护栏的实施是通过多种技术的结合来实现的：

Guardrails are implemented through a combination of techniques:

• 提示工程：精心设计提示，引导更安全的行为。
• Prompt engineering: Structuring prompts carefully to guide safer behavior.
• 审核 API ：通过毒性和安全检测器运行输出（例如，OpenAI 审核 API）。
• Moderation APIs: Running outputs through toxicity and safety detectors (e.g., OpenAI Moderation API).
• 策略引擎：为主题、关键字或行为定义明确的允许列表和阻止列表。
• Policy engines: Defining explicit allowlists and blocklists for topics, keywords, or behaviors.
• 后生成过滤：在将不安全的响应发送给用户之前，对其进行分析、编辑或丢弃。
• Post-generation filtering: Analyzing and editing or discarding unsafe responses before sending them to users.
• 基于规则的执法：使用预设规则来检测特定违规行为。
• Rule-based enforcement: Using pre-set rules to catch specific violations.
• 检索增强型防护措施：使用 RAG 在输出前交叉验证事实或检查一致性。
• Retrieval-augmented guardrails: Using RAG to cross-validate facts or check consistency before output.

没有护栏

Without guardrails

如果没有防护措施，人工智能系统容易受到以下影响：

Without guardrails, AI systems are vulnerable to the following:

• 越狱：用户精心设计棘手的提示，迫使模型绕过限制并生成禁止的输出。
• Jailbreaking: Users craft tricky prompts to force the model to bypass restrictions and generate forbidden outputs.
• 幻觉：该模型在没有验证的情况下捏造出看似合理但却是虚假的信息。
• Hallucination: The model invents plausible but false information without validation.
• 偏见放大：该模型无意中强化了刻板印象或不公平的内容。
• Bias amplification: The model unintentionally reinforces stereotypes or unfair content.
• 安全风险：泄露敏感信息、泄露系统细节或导致有害行为。
• Security risks: Revealing sensitive information, leaking system details, or enabling harmful actions.

这些风险可能会给组织造成声誉损害、违反合规规定、损害用户利益，甚至带来法律后果。

These risks can cause reputational damage, compliance violations, user harm, and even legal consequences for organizations.

护栏解决方案的行业案例

Industry examples of guardrail solutions

以下领先的人工智能平台已经意识到需要健全的防护措施，并构建了专门的框架：

The following leading AI platforms have recognized the need for robust guardrails and built specialized frameworks:

• NVIDIA NeMo 护栏：
  • 专注于可信赖的对话式人工智能的开源工具包。
  • 支持输入过滤、输出审核、对话流程控制和基于语境的生成。
  • 允许开发者使用 YAML 文件以声明方式定义Rails ，从而强制机器人和助手的行为。
• NVIDIA NeMo guardrails:
  • Open-source toolkit focused on trustworthy conversational AI.
  • Supports input filtering, output moderation, conversation flow control, and grounded generation.
  • Allows developers to define rails declaratively using YAML files to enforce behavior across bots and assistants.
• Azure AI 提示防护：
  • 微软的企业级解决方案与 Azure OpenAI 服务集成。
  • 检测并阻止提示注入攻击、攻击性内容和越狱尝试。
  • 它既能主动筛选输入数据，又能被动调节输出结果，因此对受监管行业非常有效。
• Azure AI Prompt Shields:
  • Microsoft's enterprise-grade solution is integrated with the Azure OpenAI Service.
  • Detects and blocks prompt injection attacks, offensive content, and jailbreak attempts.
  • Provides both proactive input screening and reactive output moderation, making it highly effective for regulated industries.
• OpenAI 内容审核 API ：OpenAI 内容审核 API 是应用最广泛的自动内容安全检查工具之一。该 API 会分析用户输入和模型输出，以检测各类敏感内容，例如：
  • 恨
  • 骚扰
  • 性内容
  • 暴力
  • 自我伤害
  • 误导性信息
• OpenAI Moderation API: One of the most widely used tools for automatic content safety checks is the OpenAI Moderation API. This API analyzes both user inputs and model outputs to detect sensitive content across categories such as:
  • Hate
  • Harassment
  • Sexual content
  • Violence
  • Self-harm
  • Misleading information

审核 API 会返回详细的分数，这些分数表明违规的可能性，从而允许开发者：

The Moderation API returns detailed scores indicating the likelihood of a violation, allowing developers to:

• 自动阻止不安全的查询或代码补全。
• Automatically block unsafe queries or completions.
• 标记输出结果以供人工审核。
• Flag outputs for human review.
• 根据严重程度阈值自定义工作流程。
• Customize workflows based on severity thresholds.

通过将审核 API 集成到生产管道中，开发人员可以确保模型的行为符合安全和合规标准，而无需持续的人工监控。

By integrating the Moderation API into production pipelines, developers ensure that models behave consistently with safety and compliance standards, without requiring constant manual monitoring.

这些工具表明，防护措施不再是可有可无的，而是构建负责任的、可用于生产的 AI 应用的基础。

These tools show that guardrails are no longer optional and are foundational to building responsible, production-ready AI applications.

虽然检索、重排序和防护机制显著提高了人工智能系统的可靠性和安全性，但真正的智能行为要求模型超越单次响应。现代人工智能应用越来越多地涉及智能体。智能体是能够自主推理、决策、规划和使用工具的系统。尽管我们将探讨智能体，但……在第五章“实现具有人机交互的智能体全智能系统”中，我们将更深入地探讨这些内容。在此之前，有必要介绍一些核心概念：智能体如何利用工具、进行推理、制定计划、执行动作、跨任务保持记忆，以及如何在多智能体系统中协作解决复杂目标。理解这些基础概念将有助于我们在后续章节中构建更动态、更具适应性的人工智能解决方案。

While retrieval, reranking, and guardrails significantly enhance the reliability and safety of AI systems, true intelligent behavior requires models to go beyond single-turn responses. Modern AI applications increasingly involve agents. They are systems capable of autonomous reasoning, decision-making, planning, and tool use. Although we will explore agents in greater depth in Chapter 5, Implementing Agentic GenAI Systems with Human-AI Interaction, it is important to introduce the core concepts: how agents leverage tools, perform reasoning, develop plans, execute actions, maintain memory across tasks, and collaborate in multi-agent systems to solve complex goals. Understanding these foundational ideas will prepare us for building more dynamic, adaptable AI solutions in the chapters ahead.

代理人

Agents

生成人工智能（GenAI）智能体是一种智能软件系统，它利用生成模型（例如逻辑线性模型或扩散模型）来理解、推理并根据用户输入或环境刺激创建内容。它可以执行诸如回答问题、生成文本或图像、总结内容，甚至协作解决问题等任务。生成人工智能智能体通常与各种工具或应用程序接口（API）集成，既可以独立运行，也可以在大型多智能体系统中运行。它们观察输入，基于学习到的模式做出决策，并采取与目标一致的行动，在创造性和功能性环境中模拟人类的认知能力。请参考以下列表以更深入地了解智能体：

A GenAI agent is an intelligent software system that uses generative models, such as LLMs or diffusion models, to understand, reason, and create content in response to user input or environmental stimuli. It can perform tasks like answering questions, generating text or images, summarizing content, or even collaborating in problem-solving. GenAI agents often integrate with tools or APIs and can operate autonomously or within a larger multi-agent system. They observe inputs, make decisions based on learned patterns, and take actions aligned with their goals, mimicking human-like cognition in creative and functional contexts. Refer to the following list to build a deeper understanding of agents:

• 工具是指代理可以调用​​的外部函数、API 或实用程序，它们可以扩展代理的功能，使其超越纯文本生成。代理不再需要依靠自身知识来回答所有问题，而是可以访问搜索引擎、数据库、计算器、知识图谱或自定义 API 来收集实时信息或执行特定操作。工具使代理更加强大，能够与外部世界交互、检索最新信息、查询私有数据或执行仅靠模型无法可靠完成的任务。
• Tools are external functions, APIs, or utilities that an agent can call upon to extend its capabilities beyond pure text generation. Instead of trying to answer all questions from its own knowledge, an agent can access search engines, databases, calculators, knowledge graphs, or custom APIs to gather real-time information or perform specific operations. Tools make agents more powerful, enabling them to interact with the external world, retrieve up-to-date facts, query private data, or execute tasks that a model alone could not accomplish reliably.
• 推理是指智能体逐步思考问题的能力，而不是立即生成输出。通过推理，智能体评估当前情况，确定已知信息，识别需要了解的信息，并选择一系列行动以逐步接近任务目标。推理使智能体能够将复杂的目标分解为易于处理的子问题，应对不确定性，并根据新信息或意外结果动态调整自身行为。
• Reasoning refers to the agent’s ability to think through problems step-by-step, rather than immediately generating an output. Through reasoning, an agent evaluates the current situation, decides what it knows, identifies what it needs to find out, and chooses a sequence of actions that move it closer to solving the task. Reasoning allows agents to break down complex goals into manageable subproblems, handle uncertainty, and adapt their behavior dynamically based on new information or unexpected results.

• 规划建立在推理之上，涉及为实现既定目标而对未来步骤进行结构化组织。一个设计良好的智能体并非只是被动地一步一步做出反应，而是能够构建一个灵活的、目标导向的计划——一系列精心设计的工具调用、决策和行动，最终导向成功的结果。高级智能体通过自我反思、评估中间步骤并在必要时调整策略以及CoT提示等技术来增强其规划能力。CoT提示鼓励智能体在执行前进行系统性的多步骤推理。这使得智能体不仅能够设定中间目标并确定任务优先级，还能在面对意外结果或信息不完整时动态地进行即兴发挥。有效的规划使智能体能够高效地应对复杂的多阶段流程，而不是冲动行事或被早期错误所困。规划、反思、逻辑推理和实时适应的能力对于解决需要在多个行动和决策点上持续、连贯地推进的现实世界任务至关重要。
  • 动作代表执行阶段，在此阶段，智能体执行其计划中的特定步骤，例如调用外部工具、发出 API 请求、将​​信息保存到内存或向用户返回答案。动作是智能体系统的执行部分，在此阶段，推理和规划转化为可观察的操作。重要的是，动作可以根据先前动作的结果动态变化；智能体可以修改其计划，再次进行推理，并采取不同的后续步骤。这种持续的思考和行动循环使智能体区别于静态模型。
  • 记忆和多智能体协作是进一步增强智能体能力的先进概念。记忆使智能体能够在不同的步骤、会话甚至任务中保留信息，从而实现连续性、个性化和长期学习。拥有记忆功能的智能体无需每次都从头开始，而是可以回忆过去的交互、中间结果和不断变化的用户偏好。多智能体协作进一步扩展了这一功能：多个专业化的智能体可以协同工作，共享任务、分配职责并相互沟通，从而比单个智能体更高效地解决复杂目标。具备记忆和协作能力的系统开始类似于协调的、模块化的智能体生态系统，这些智能体共同朝着共同的目标努力。下图展示了智能体如何与其环境交互并采取行动：
• Planning builds upon reasoning and involves the structured organization of future steps to achieve a given objective. A well-designed agent does not simply react one step at a time but is capable of constructing a flexible, goal-directed plan—a deliberate sequence of tool calls, decisions, and actions that lead to a successful outcome. Advanced agents enhance their planning through techniques like self-reflection, evaluating their intermediate steps and adjusting strategies if needed, and CoT prompting, which encourages systematic, multi-step reasoning before execution. This enables agents not only to set intermediate goals and prioritize tasks but also to improvise dynamically when faced with unexpected results or incomplete information. Effective planning allows agents to navigate complex, multi-stage processes efficiently rather than acting impulsively or getting trapped by early errors. The ability to plan, reflect, reason through chains of logic, and adapt in real-time is critical for solving real-world tasks that require sustained, coherent progress across multiple actions and decision points.
  • Action represents the execution phase where an agent implements a specific step in its plan, such as calling an external tool, making an API request, saving information to memory, or returning an answer to the user. Actions are the doing part of agentic systems, where reasoning and planning turn into observable operations. Importantly, actions can be dynamic based on the results of previous actions; the agent may revise its plan, reason again, and take different subsequent steps. This continuous loop of thinking and acting distinguishes agents from static models.
  • Memory and multi-agent collaboration are advanced concepts that further enhance an agent’s capabilities. Memory allows agents to retain information across different steps, sessions, or even tasks, providing continuity, personalization, and long-term learning. Instead of starting from scratch every time, an agent with memory can recall past interactions, intermediate results, and evolving user preferences. Multi-agent collaboration expands this even further: multiple specialized agents can work together, sharing tasks, delegating responsibilities, and communicating with each other to solve complex goals more efficiently than a single agent could. Systems with memory and collaboration capabilities begin to resemble coordinated, modular ecosystems of intelligent agents working towards shared objectives. The following figure illustrates how an agent interacts with its environment and takes action:

主动型 RAG 与非主动型 RAG

Agentic RAG vs. non-agentic RAG

在非智能体 RAG 系统中，流程是线性且静态的：用户嵌入查询，检索器获取前 k 个文档，语言模型利用检索到的上下文生成答案。每个步骤都遵循固定的流程，缺乏动态决策。非智能体 RAG 在简单的问答任务中表现出色，因为初始检索结果通常足够，但当检索结果嘈杂、模糊或不足以进行复杂推理时，它就显得力不从心。

In a non-agentic RAG system, the process is linear and static: a user query is embedded, a retriever fetches top-k documents, and the language model-generates an answer using the retrieved context. Each step follows a fixed pipeline without dynamic decision-making. Non-agentic RAG excels in simple question answer tasks where the initial retrieval is usually sufficient, but it struggles when retrieval results are noisy, ambiguous, or insufficient for complex reasoning.

相比之下，智能体 RAG 系统引入了动态控制、推理和适应性。智能体首先评估查询，检索初始文档，并判断信息是否充足。如果信息不足，智能体可以重新构建查询，执行多次检索，选择不同的工具（例如搜索 API 或数据库），反思中间结果，并动态规划多个步骤以得出更可靠的答案。智能体 RAG 系统可以迭代地从多个知识源检索、重新排序、推理和综合信息，并实时调整以解决复杂、多跳或歧义的查询。

In contrast, an agentic RAG system introduces dynamic control, reasoning, and adaptability. An agent first assesses the query, retrieves initial documents, and reasons about whether the information is sufficient. If not, the agent can reformulate the query, perform multiple retrievals, choose different tools (like search APIs or databases), reflect on intermediate results, and dynamically plan multiple steps to arrive at a better-grounded answer. Agentic RAG systems can iteratively retrieve, rerank, reason, and synthesize across multiple knowledge sources, adapting in real-time to solve complex, multi-hop, or ambiguous queries.

因此，虽然非智能体 RAG 对于简单的任务来说简单快捷，但智能体 RAG 对于构建真正智能、可靠的系统至关重要，这些系统能够处理不确定性、不完整的数据或不断变化的信息需求。图 1.13展示了一个多智能体系统，该系统包含一个编排智能体和两个能够执行联合任务以及单独任务的附加智能体：

Thus, while non-agentic RAG is simple and fast for straightforward tasks, agentic RAG is critical for building truly intelligent, reliable systems that can handle uncertainty, incomplete data, or evolving information needs. Figure 1.13 illustrates a multi-agent system featuring an orchestration agent and two additional agents capable of performing joint tasks as well as individual tasks:

模型上下文协议

Model Context Protocols

智能体系统赋予人工智能模型自主推理、规划和执行任务的能力，其方式是通过动态地使用工具、API 和外部知识库。然而，如果没有标准化的方法来发现和交互这些工具，扩展就会变得混乱且脆弱。这正是 MCP 的关键所在。MCP 为智能体提供了一个通用的、与语言无关的接口，使其能够无缝访问工具、数据和提示，从而确保安全、模块化和动态的集成。

Agentic systems empower AI models to reason, plan, and execute tasks autonomously by dynamically using tools, APIs, and external knowledge sources. However, without a standardized way to discover and interact with these tools, scaling becomes chaotic and fragile. This is where the MCP is essential. MCP provides a universal, language-agnostic interface for agents to seamlessly access tools, data, and prompts, ensuring secure, modular, and dynamic integration.

MCP 是一种开放标准，旨在简化和规范 AI 模型与外部工具、数据源和 API 的交互方式。MCP 由Anthropic公司推出，它充当通用通信层，类似于 AI 领域的 USB-C 接口，使 AI 助手和代理能够无缝检索结构化信息、调用操作或应用特定领域的提示，而无需为每个后端系统进行自定义集成。

MCP is an open standard designed to simplify and standardize how AI models interact with external tools, data sources, and APIs. Introduced by Anthropic, MCP acts as a universal communication layer, much like a USB-C for AI, enabling AI assistants and agents to seamlessly retrieve structured information, invoke actions, or apply domain-specific prompts without custom integrations for every backend system.

MCP 的核心是构建客户端-服务器架构，其中服务器公开三种基本要素：工具（执行操作的函数）、资源（数据，例如文档或 API）和提示（AI 行为的指导）。MCP 使用轻量级、与语言无关的协议，例如 JSON-RPC，并通过 Studio 或 HTTP/SSE 等传输协议进行通信，因此易于在各种环境中集成。

At its core, MCP establishes a client-server architecture where servers expose three primitives: tools (functions that perform actions), resources (data like documents or APIs), and prompts (guidance for AI behavior). MCP uses lightweight, language-agnostic protocols like JSON-RPC over transports such as studio or HTTP/SSE, making it easy to integrate across diverse environments.

通过采用 MCP，开发人员可以构建可扩展的 AI 系统，无需重新训练模型或硬编码 API，即可动态发现和利用新的工具和数据源。MCP 还确保了模块化、安全性和面向未来的适应性，这对于医疗保健、金融和企业自动化等行业至关重要。随着 AI 生态系统日益复杂，MCP 为构建可互操作、安全且敏捷的 AI 系统奠定了基础，这些系统可以通过统一的接口在多个领域进行推理和行动。

By adopting MCP, developers can build scalable AI systems where new tools and data sources can be dynamically discovered and utilized without retraining models or hardcoding APIs. MCP also ensures modularity, security, and future-proofing, critical for sectors like healthcare, finance, and enterprise automation. As AI ecosystems grow increasingly complex, MCP provides a foundation for building interoperable, secure, and agile AI systems that can reason and act across multiple domains through a unified interface.

智能体系统与 MCP 的结合，使人工智能能够更智能、更可靠地运行，无需硬编码依赖即可适应现实世界的复杂性，从而在医疗保健、金融和教育等行业中释放强大的应用潜力。图 1.14展示了 MCP 如何建立客户端-服务器架构并与外部工具、数据源和 API 进行交互：

Together, agentic systems and MCP enable AI to operate more intelligently and reliably, adapting to real-world complexities without hardcoded dependencies, unlocking powerful applications across industries like healthcare, finance, and education. Figure 1.14 shows how MCP establishes a client-server architecture and interacts with external tools, data sources, and APIs:

结论

Conclusion

本章为理解现代生成人工智能（GenAI）系统的设计和编排奠定了基础。我们首先区分了检索系统和生成系统，探讨了它们各自在构建智能人工智能解决方案中发挥的关键作用。我们讨论了从传统的基于关键词的检索到由嵌入驱动的密集向量搜索的演变，以及向量数据库如何实现可扩展的实时语义检索。超越基础检索，我们介绍了重排序技术，特别是交叉编码器的应用，以优化检索到的文档并确定其优先级，从而提高相关性和精确度。随后，我们强调了安全防护措施的重要性，以确保人工智能的输出安全、符合伦理，并符合现实世界的使用标准。最后，我们介绍了新兴的智能体人工智能系统，涵盖了工具使用、推理、规划、行动、记忆和多智能体协作等关键概念。

In this chapter, we laid the foundation for understanding how modern GenAI systems are designed and orchestrated. We began by differentiating between retrieval systems and generation systems, exploring how each plays a critical role in building intelligent AI solutions. We discussed the evolution from traditional keyword-based retrieval to dense vector search powered by embeddings, and how vector databases enable scalable, real-time semantic retrieval. Moving beyond basic retrieval, we introduced reranking techniques, particularly the use of cross-encoders, to refine and prioritize retrieved documents for greater relevance and precision. We then emphasized the importance of guardrails to ensure AI outputs are safe, ethical, and aligned with real-world usage standards. Finally, we introduced the emerging world of agentic AI systems, covering key concepts such as tool use, reasoning, planning, action, memory, and multi-agent collaboration.

下一章，我们将探索不断扩展的多模态系统领域，人工智能应用不再局限于单一的输入或输出模式。重点将转向多模态GenAI架构，在这种架构中，文本、图像和结构化数据在统一的框架内进行交互。读者将学习人工智能系统如何将文本转换为图像，如何将图像解读为描述，如何组合输入以生成新的输出，甚至如何将自然语言翻译成结构化查询语言（SQL ）。这为构建丰富且具有上下文感知能力的人工智能体验奠定了基础。

In the next chapter, we explore the expanding frontier of multimodal systems, where AI applications are no longer limited to a single mode of input or output. The focus then shifts to multimodal GenAI architectures, where text, images, and structured data interact within unified frameworks. Readers will learn how AI systems transform text into images, interpret images into descriptions, combine inputs for new outputs, and even translate natural language into Structured Query Language (SQL). This sets the foundation for building rich, contextually aware AI experiences.

介绍

Introduction

第一章介绍了现代生成式人工智能（GenAI ）的基础知识，涵盖检索系统、生成模型、检索增强生成（RAG ）、编排、分词、向量数据库、重排序、防护机制、代理系统和模型上下文协议（MCP ）。这些核心组件为构建智能的、文本驱动的生成系统奠定了基础。

The first chapter introduced the foundations of modern generative AI (GenAI), covering retrieval systems, generation models, retrieval-augmented generation (RAG), orchestration, tokenization, vector databases, reranking, guardrails, agent systems, and Model Context Protocols (MCPs). These core components established the groundwork for building intelligent, text-driven generative systems.

在此基础上，本章探讨了人工智能向多模态领域的演进，即文本、图像和其他数据类型的协同处理。我们首先在视觉语言模型（VLM ）的背景下解释交叉编码器和双编码器，然后讨论多模态向量嵌入和多模态向量数据库的设计。

Building on this foundation, this chapter explores the evolution of AI into multimodal domains, where text, images, and other data types are processed together. We begin by explaining cross-encoders and bi-encoders within the context of vision-language models (VLMs), followed by a discussion on multimodal vector embeddings and the design of multimodal vector databases.

本章进一步阐明了虚拟语言模型（VLM）与更广泛的多模态全息人工智能（GenAI）系统的区别。文中涵盖了实际应用，包括文本到图像生成、图像到文本的描述、文本和图像到图像的合成，以及基于文本的规范和图像生成。此外，我们还探讨了文本到SQL查询生成如何扩展多模态人工智能系统的潜力。

The chapter further clarifies how VLMs differ from broader multimodal GenAI systems. Practical applications, including text-to-image generation, image-to-text captioning, text and image-to-image synthesis, and text-driven specification and image generation, are covered. Additionally, we explore how text-to-SQL query generation expands the potential of multimodal AI systems.

本章中，我们将从理解生成模型的基本机制，逐步过渡到开发能够进行复杂、跨模态推理的系统，从而为在现实世界环境中进行高级应用做好准备。

Through this chapter, we move from understanding the basic mechanisms of generative models to developing systems capable of sophisticated, cross-modal reasoning, positioning us for advanced applications in real-world environments.

结构

Structure

本章我们将学习以下主题：

In this chapter, we will learn about the following topics:

• 理解视觉语言模型
• Understanding vision-language model
• 实施方案比较
• Implementation comparisons
• 多模态生成式人工智能系统与虚拟语言模型
• Multimodal generative AI systems vs. VLMs
• 视觉语言模型
• Vision-language models
• 基于输出的多模态系统分类
• Output-based classification of multimodal systems

目标

Objectives

本章旨在帮助读者全面理解设计和部署现代全智能（GenAI）系统所需的关键构建模块。通过掌握检索和生成系统、向量数据库、嵌入技术、高级提示策略、智能体架构和多智能体协作等概念，读者将为构建智能、可扩展的人工智能解决方案奠定坚实的基础。此外，本章还介绍了本地模型部署、图形处理器（GPU ）基础设施、语音处理、智能体内存管理以及多智能体协作（MCP）等行业标准等关键主题。这些基础要素对于构建多模态、可靠且可用于生产环境的人工智能应用至关重要。

This chapter aims to equip readers with a comprehensive understanding of the key building blocks essential for designing and deploying modern GenAI systems. By mastering concepts such as retrieval and generation systems, vector databases, embedding techniques, advanced prompting strategies, agentic architectures, and multi-agent collaboration, readers will gain a strong foundation for building intelligent, scalable AI solutions. Additionally, the chapter introduces critical topics like local model deployment, graphics processing unit (GPU) infrastructure, speech processing, memory management in agents, and industry standards like MCPs. These foundational elements are crucial for advancing toward multimodal, reliable, and production-ready AI applications.

理解视觉语言模型

Understanding vision-language models

视觉语言模型（VLM）是多模态人工智能系统的基础，它弥合了视觉和文本理解之间的鸿沟。与主要独立处理文本或图像的传统全息人工智能（GenAI）不同，VLM旨在跨模态进行联合解释、对齐和生成。随着越来越多的组织寻求构建能够“看”和“说”的系统，VLM对于视觉问答（VQA ）、图像字幕生成、跨模态检索，甚至文本驱动的图像生成等应用都变得至关重要。

VLMs form the foundation of multimodal AI systems that bridge the gap between visual and textual understanding. Unlike traditional GenAI, which primarily processes text or images in isolation, VLMs are designed to jointly interpret, align, and generate across both modalities. As organizations increasingly look to create systems that see and talk, VLMs have become critical for applications such as visual question answering (VQA), captioning, cross-modal retrieval, and even text-driven image generation.

在前一章讨论核心生成和检索概念的基础上，本节深入探讨了 VLM 的架构、类型和功能，重点介绍了它们如何将分词、向量嵌入、检索和生成原理扩展到多种数据形式。

Building on the foundations laid in the previous chapter, where we discussed core generative and retrieval concepts, this section delves into the architecture, types, and capabilities of VLMs, highlighting how they extend the principles of tokenization, vector embeddings, retrieval, and generation across multiple data forms.

视觉语言模型的分类

Categories of vision-language models

视觉语言模型（VLM）是功能强大的AI系统，它融合了视觉和文本理解能力，使机器能够处理、解释和生成跨两种模态的信息。随着该领域的不断发展，VLM的应用范围日益广泛，从检索与搜索查询匹配的图像到生成详细的图像描述，甚至对文档进行推理。为了更好地理解VLM的功能和设计，我们可以对其进行广泛的分析。根据其核心目标，虚拟语言模型（VLM）可分为以下几类：检索、字幕生成和质量保证（QA）、生成式合成、多模态推理和指令调整。每一类都体现了独特的架构重点，并支持电子商务、无障碍访问、教育和设计等行业的实际应用。VLM 可以根据其主要任务和架构设计目标进行大致分类，具体如下：

VLMs are powerful AI systems that integrate visual and textual understanding, enabling machines to process, interpret, and generate information across both modalities. As the field evolves, VLMs are increasingly specialized in serving diverse applications, from retrieving the right image for a search query to generating detailed image descriptions or even reasoning over documents. To better understand their capabilities and design, VLMs can be broadly categorized based on their core objectives: retrieval, captioning, and QA, generative synthesis, multimodal reasoning, and instruction-tuning. Each category reflects a unique architectural focus and supports real-world applications across industries like e-commerce, accessibility, education, and design. VLMs can be broadly classified based on their primary tasks and architectural design goals, which are as follows:

• 面向检索的虚拟语言模型：这类模型经过训练，能够将图像和文本对齐到共享的嵌入空间，从而实现高效的跨模态检索。其关键目标是找到与文本查询最相关的图像（反之亦然）：
  • CLIP（OpenAI）：经过 4 亿个图像-文本对的训练；它学习将图像及其相关的文本描述紧密地嵌入到潜在空间中。
  • ALIGN（谷歌）：将检索范式扩展到来自网络的数十亿个带噪声的图像-文本对。
  • DeCLIP ：通过考虑更难的负样本和更好的对比损失，改进了对比语言-图像预训练（CLIP ）。
  • 应用领域：图像搜索引擎、内容审核、电子商务产品发现。
• Retrieval-focused VLMs: These models are trained to align images and text into a shared embedding space, allowing efficient cross-modal retrieval. A key objective is to find the most relevant image given by a text query (or vice versa):
  • CLIP (OpenAI): Trained on 400 million image-text pairs; it learns to embed images and their associated text descriptions closely in the latent space.
  • ALIGN (Google): Scales the retrieval paradigm to billions of noisy image-text pairs from the web.
  • DeCLIP: Improves over Contrastive Language–Image Pretraining (CLIP) by considering harder negatives and better contrastive losses.
  • Applications: Image search engines, content moderation, e-commerce product discovery.
• VQA和图像字幕 VLM ：这些模型旨在回答有关图像的问题或生成描述图像的字幕：
  • 从 Transformer 学习跨模态编码器表示 (LXMERT) (Facebook)：LXMERT通过跨模态转换器合并单独的视觉和语言表示。
  • 通用图像-文本表示学习 (UNITER) (微软)： UNITER训练视觉和文本的统一表示，取得了强大的 VQA 结果。
  • VisualBERT ：它将视觉区域特征早期融入到Transformers 的双向编码器表示( BERT ) 文本转换器中，从而改进字幕生成和 VQA。
  • 应用：辅助工具（例如，为视障人士提供的图像描述）、视觉助手和内容标记。
• VQA and captioning VLMs: These models are designed to answer questions about images or generate captions that describe images:
  • Learning Cross-Modality Encoder Representations from Transformers (LXMERT) (Facebook): LXMERT merges separate visual and language representations via a cross-modal transformer.
  • UNiversal Image-TExt Representation Learning (UNITER) (Microsoft): UNITER trains a unified representation for both vision and text, achieving strong VQA results.
  • VisualBERT: It incorporates visual region features early into Bidirectional Encoder Representations from Transformers (BERT) text transformer, improving captioning and VQA.
  • Applications: Accessibility tools (e.g., image descriptions for the visually impaired), visual assistants, and content tagging.
• 生成式虚拟语言模型：这些模型可以根据跨模态的输入生成新的图像或详细文本：
  • DALL·E 2 (OpenAI) ：根据文本提示生成逼真的图像。
  • Imagen（谷歌）：最先进的文本到图像生成模型，专注于超高质量图像。
  • Parti（谷歌）：通过对视觉标记序列进行建模来生成图像的自回归模型。
  • 应用领域：创意设计、游戏开发、营销视觉、虚拟世界。
• Generative VLMs: These models generate new images or detailed text from inputs across modalities:
  • DALL·E 2 (OpenAI): Generates photorealistic images from text prompts.
  • Imagen (Google): State-of-the-art text-to-image generation model, focusing on ultra-high-quality images.
  • Parti (Google): Autoregressive model generating images by modeling sequences of visual tokens.
  • Applications: Creative design, game development, marketing visuals, virtual worlds.
• 多模态推理 VLM ：这些模型超越了检索或生成，能够跨模态执行逻辑推理，例如推断图像和文本之间的关系，或解决复杂的视觉文本任务：
  • Flamingo（DeepMind）：对图像和文本进行灵活条件反射的少样本视觉推理。
  • Kosmos-1（微软）：将多模态推理扩展到文档、图像、光学字符识别( OCR ) 文本，甚至视觉数学问题。
  • 应用领域：多模态聊天机器人、文档理解和教育技术。
• Multimodal reasoning VLMs: These models go beyond retrieval or generation, performing logical reasoning across modalities, such as inferring relationships between images and text, or solving complex visual-text tasks:
  • Flamingo (DeepMind): Few-shot visual reasoning with flexible conditioning on images and text.
  • Kosmos-1 (Microsoft): Extends multimodal reasoning to documents, images, Optical Character Recognition (OCR) text, and even visual math problems.
  • Applications: Multimodal chatbots, document understanding, and education technologies.
• 指令调整的虚拟语言模型：以下最新模型使用类似人类的指令进行调整，提高了它们在无需显式重新训练的情况下泛化到新任务的能力：
  • BLIP（Salesforce）：从图像描述和检索中进行引导学习；既可以生成图像描述，也可以检索图像。
  • BLIP-2（Salesforce）：将视觉编码器与冻结的语言模型（例如 OPT、Flan-T5）连接起来，用于零样本 VQA 和生成。
  • MiniGPT-4 ：一种构建类似 MiniGPT-4 的多模态模型的轻量级开源方法。
  • 应用领域：零样本图像标注、面向任务的多模态系统、机器人感知。
• Instruction-tuned VLMs: The following recent models are aligned using human-like instructions, improving their ability to generalize across new tasks without explicit retraining:
  • BLIP (Salesforce): Bootstrap learning from captioning and retrieval; can both generate captions and retrieve images.
  • BLIP-2 (Salesforce): Connects a vision encoder with a frozen language model (e.g., OPT, Flan-T5) for zero-shot VQA and generation.
  • MiniGPT-4: Lightweight, open-source approach to build a MiniGPT-4-like multimodal model.
  • Applications: Zero-shot captioning, task-oriented multimodal systems, robotic perception.

视觉语言模型的核心架构组件

Core architectural components of vision-language models

尽管任务各不相同，但大多数虚拟学习模块都遵循一些共同的设计原则，例如：

Despite task differences, most VLMs share common design principles, like the following:

• 交叉编码器与双向编码器：
  • 交叉编码器在推理过程中联合处理图像和文本，从而实现更丰富的交互，但检索速度较慢。它将图像和文本输入一起接收，并通过一个联合的Transformer模型进行处理。图像通常表示为一系列视觉标记（例如，来自视觉Transformer ( ViT ) 或具有空间展平的卷积神经网络( CNN )），而文本则按常规方式进行标记化。

    然后将这两个序列连接起来，并传递给一个转换器，该转换器对整个连接序列执行自注意力机制，这意味着：

    • 文本标记关注图像标记。
    • 图像标记关注文本标记。
    • 令牌在其模态内也会相互关注。

    这种完全的交叉注意力机制使得视觉和语言表征在每一层都能进行丰富而精细的交互。例如，像“狗”这样的词可以关注显示狗的特定图像块，反之亦然。

    图 2.1展示了一种基于交叉编码器的 VLM 架构，该架构专为需要深度理解图像及其对应文本描述的任务而设计，例如将产品照片与详细规格进行匹配。与为图像和文本分别生成嵌入向量的双编码器不同，该方法不依赖于嵌入向量，而是通过将输入作为组合对进行处理来计算直接相关性得分。通过融合注意力机制或交叉注意力机制，该模型能够捕捉跨模态的细粒度交互。这种设置非常适合对对齐精度要求极高的场景，例如电子商务产品验证、视频问答 (VQA) 和多模态文档理解。

• Cross-encoders vs. bi-encoders:
  • Cross-encoders jointly process both image and text during inference, allowing richer interaction but slower retrieval. It takes both the image and text input together and processes them through a single, joint transformer model. The image is usually represented as a sequence of visual tokens (e.g., from a Vision Transformer (ViT) or a convolutional neural network (CNN) with spatial flattening), and the text is tokenized as usual.

    These two sequences are then concatenated and passed into a transformer that performs self-attention across the entire joint sequence, meaning:

    • Text tokens attend to image tokens.
    • Image tokens attend to text tokens.
    • Tokens attend to each other within their modality as well.

    This full cross-attention allows rich, fine-grained interactions between vision and language representations at every layer. For example, a word like dog can attend to specific image patches showing the dog, and vice versa.

    Figure 2.1 depicts a cross-encoder-based VLM architecture designed for tasks requiring deep joint understanding of images and their corresponding textual descriptions, such as matching product photos with detailed specifications. Unlike dual encoders that generate separate embeddings for images and text, this approach does not rely on embeddings but instead computes a direct relevance score by processing the input as a combined pair. Through merged attention or cross-attention mechanisms, the model captures fine-grained interactions across modalities. This setup is ideal for scenarios where alignment precision is critical, such as e-commerce product verification, VQA, and multimodal document understanding.

• 双编码器分别对图像和文本进行编码，然后比较它们的嵌入，从而实现更快、可扩展的检索（如 CLIP 中所用）。

  图像和文本分别使用两个独立的编码器进行编码，生成各自的嵌入向量。通常情况下，视觉编码器（例如，ViT 或 CNN）处理图像并生成图像嵌入向量；文本编码器处理文本嵌入向量。（例如，基于Transformer的语言模型，如BERT）处理文本并生成文本嵌入向量。

  重要的是，图像和文本在编码过程中不会相互干扰。

  在推理过程中，图像特征和文本特征之间不存在交叉注意力。

  分别生成两个嵌入向量后，在编码后对它们进行比较，通常通过计算相似度得分来实现，例如：

  • 余弦相似度
  • 点积
  • 欧氏距离

  下图展示了两个嵌入在向量空间中越接近，它们之间的相关性就越高：

• Bi-encoders encode image and text separately and compare their embeddings later, enabling faster, scalable retrieval (as used in CLIP).

  The image and text are encoded separately into their own embeddings using two independent encoders, typically: a vision encoder (e.g., a ViT or CNN) processes the image and produces an image embedding vector. A text encoder (e.g., a transformer-based language model like BERT) processes the text and produces a text embedding vector.

  Importantly, the image and text do not interact during encoding.

  There is no cross-attention between image and text features during inference.

  Once both embeddings are generated independently, they are compared after encoding, often by computing a similarity score such as:

  • Cosine similarity
  • Dot product
  • Euclidean distance

  The following figure depicts how closer the two embeddings are in the vector space, the more relevant they are considered to each other:

• 融合机制：一些模型（如 LXMERT、VisualBERT）在模型训练期间使用早期或晚期融合来结合视觉和文本信息。

  在视觉语言模型（VLM）中，融合机制指的是如何将来自不同模态的信息（通常是图像的视觉特征和语言的文本特征）结合起来，形成联合表示。有效的融合对于使模型能够处理视觉和文本输入至关重要。

• Fusion mechanisms: Some models (like LXMERT, VisualBERT) use early or late fusion to combine visual and textual information during model training.

  Fusion mechanisms in VLMs refer to how information from different modalities, typically visual features from images and textual features from language, is combined to form a joint representation. Effective fusion is crucial for enabling models to reason across both vision and text inputs.

融合策略有多种类型。早期融合在输入阶段将图像和文本嵌入结合起来，使模型能够从一开始就联合学习跨模态交互。后期融合则分别处理每种模态，并在后期将它们的输出合并。中间融合阶段通常在最终决策之前进行。中间融合（或跨模态融合）在部分处理后组合特征，从而在模型的前向传播过程中实现更复杂的模态间交互。融合机制通常使用交叉注意力层来实现，其中来自一种模态（例如图像区域）的特征会关注来自另一种模态（例如文本标记）的特征。这类似于Transformer模型使用注意力来关联序列的不同部分，但这里的注意力是跨模态的。交叉注意力使模型能够在处理文本时选择性地关注图像的相关部分，反之亦然。

There are several types of fusion strategies. Early fusion combines image and text embeddings at the input stage, allowing the model to jointly learn cross-modal interactions from the beginning. Late fusion processes each modality separately and merges their output at a later stage, typically before final decision-making. Intermediate fusion (or cross-modal fusion) combines features after partial processing, allowing for more sophisticated interactions between modalities during the model's forward pass. Fusion mechanisms are often implemented using cross-attention layers, where features from one modality (e.g., image regions) attend to features from the other modality (e.g., text tokens). This is similar to how transformers use attention to relate different parts of a sequence, but here the attention operates across modalities. Cross-attention enables models to selectively focus on relevant parts of an image when processing text and vice versa.

因此，虽然融合机制广义上是指模态的组合，但交叉注意力是一种经常在融合策略中使用的具体技术。

Thus, while fusion mechanisms refer broadly to the combining of modalities, cross-attention is a specific technique often used within fusion strategies.

视觉学习模型（VLM）代表了人工智能的一次关键演进，它将计算机视觉和自然语言处理（NLP ）的优势融合到统一而强大的架构中。从检索和图像描述到多模态推理和指令调优，VLM 正在为下一代智能系统铺平道路，这些系统能够通过多种感官与世界互动。在接下来的章节中，我们将深入探讨多模态全人类人工智能（GenAI）系统，VLM 的能力和局限性将提供一个重要的参考点，凸显构建真正多功能、类人人工智能的机遇和挑战。

VLMs represent a critical evolution in AI, merging the strengths of computer vision and natural language processing (NLP) into unified, powerful architectures. From retrieval and captioning to multimodal reasoning and instruction-tuning, VLMs are paving the way for the next-generation of intelligent systems capable of interacting with the world through multiple senses. As we proceed deeper into multimodal GenAI systems in the next sections, the capabilities and limitations of VLMs provide a vital reference point, highlighting the opportunities and challenges of building truly versatile, human-like AI.

视觉语言模型面临的挑战

Challenges in vision-language models

尽管虚拟语言模型在视觉质量评估、图像描述和跨模态检索等任务中取得了越来越大的成功，但它们仍面临着一些关键挑战，限制了其更广泛的应用和实际部署。这些挑战既源于架构上的局限性，也源于数据、性能和泛化方面的实际限制：

Despite their growing success across tasks such as VQA, image captioning, and cross-modal retrieval, VLMs face several critical challenges that limit their broader applicability and real-world deployment. These challenges stem from both architectural limitations and practical constraints in data, performance, and generalization:

• 数据需求和质量：视觉学习模型（VLM）需要大量的对齐图像-文本数据来学习有意义的跨模态表征。COCO、Visual Genome 或 LAION 等高质量数据集提供了一个起点，但它们通常偏向于西方、以互联网为中心的内容。这限制了模型在特定领域或非英语环境中的泛化能力。此外，大规模创建精心整理、对齐良好的图像-文本对成本高昂，而且噪声字幕会降低模型学习效果。
• Data requirements and quality: VLMs require vast amounts of aligned image-text data to learn meaningful cross-modal representations. High-quality datasets like COCO, Visual Genome, or LAION provide a starting point, but they are often biased toward Western, internet-centric content. This restricts generalization to domain-specific or non-English environments. Moreover, creating curated, well-aligned image-text pairs at scale is expensive, and noisy captions can degrade model learning.
• 模态失衡：在许多情况下，由于文本模态语义丰富，它在模型学习中占据主导地位，导致视觉信号利用不足。这种失衡降低了视觉-文本融合的有效性，并导致在需要精细视觉理解的任务（例如物体定位或场景描述）上表现欠佳。
• Modality imbalance: In many cases, the textual modality dominates model learning due to its semantic richness, leading to underutilization of the visual signal. This imbalance reduces the effectiveness of vision-text fusion and results in suboptimal performance on tasks requiring fine-grained visual understanding, such as object grounding or scene description.
• 多模态推理能力有限：虽然视觉学习模型能够对齐文本和图像特征，但它们跨模态进行逻辑推理的能力仍然较弱。它们难以处理需要时间理解、数值推理或多步骤推理的任务，尤其是在需要同时从视觉元素和文本上下文中提取信息时。
• Limited multimodal reasoning: While VLMs can align text and image features, their ability to perform logical reasoning across modalities remains weak. They struggle with tasks requiring temporal understanding, numerical reasoning, or multi-step inference, especially when information must be drawn jointly from visual elements and textual context.
• 跨领域泛化能力：大多数虚拟语言模型（VLM）都是在网络规模的通用数据集上训练的，因此难以泛化到特定领域，例如医学影像、科学文献或工业环境。微调有所帮助，但这通常需要目标领域的大型标注数据集，而这些数据集可能并不容易获取。
• Generalization across domains: Most VLMs are trained on web-scale and generic datasets and fail to generalize to specialized domains, such as medical imaging, scientific literature, or industrial settings. Fine-tuning helps, but it often requires large labeled datasets in the target domain, which may not be readily available.
• 效率和延迟：视觉语言模型（VLM）的训练和部署都需要耗费大量的计算资源。采用视觉和文本标记交叉注意力机制的架构（例如 UNITER、LXMERT）在推理阶段的扩展性较差。这限制了它们在低延迟或资源受限环境（例如移动设备或边缘计算）中的应用。
• Efficiency and latency: VLMs are computationally expensive to train and deploy. Architectures that use cross-attention between vision and text tokens (e.g., UNITER, LXMERT) scale poorly at inference time. This limits their feasibility in low-latency or resource-constrained environments, such as mobile devices or edge computing.
• 缺乏工具集成：与模块化的 GenAI 系统不同，大多数 VLM 并非设计用于与数据库、API 或向量存储等外部工具对接。这限制了它们在需要上下文关联或外部内存访问的动态环境中的实用性。
• Lack of tool integration: Unlike modular GenAI systems, most VLMs are not designed to interface with external tools, like databases, APIs, or vector stores. This limits their utility in dynamic environments where contextual grounding or external memory access is required.

克服这些挑战需要在数据管理、模型架构、训练效率以及与检索或编排系统的集成方面进行创新，其中许多挑战在更广泛的多模态 GenAI 框架中得到了解决。

Overcoming these challenges requires innovations in data curation, model architecture, training efficiency, and integration with retrieval or orchestration systems, many of which are addressed in broader multimodal GenAI frameworks.

多模态GenAI系统

Multimodal GenAI system

训练视觉学习模型（VLM）是一个极其消耗资源的过程。这些模型需要数百万甚至数十亿个对齐的图像-文本对才能学习有意义的多模态表示。整理如此庞大的数据集需要付出大量努力，包括数据收集、清洗、质量过滤，有时还需要人工标注以确保正确对齐。除了数据成本之外，计算成本也非常高。VLM 通常使用大型架构，例如用于图像的 ViT 和用于文本的基于 Transformer 的编码器。从头开始训练它们需要运行数周甚至数月的大量 GPU 或 TPU 集群。例如，像 CLIP（OpenAI）和 ALIGN（Google）这样的模型，由于硬件、存储和能源成本的限制，普通组织很难复制其训练数据集。此外，要实现良好的泛化能力，需要涵盖广泛的视觉和文本概念的多样化数据集，这进一步增加了数据采集的难度。对于大多数组织来说，微调预训练的 VLM 更为可行，但如果需要大规模的领域自适应，即使是这种方法也可能成本高昂。

Training VLMs is an extremely resource-intensive process. These models require millions or even billions of aligned image-text pairs to learn meaningful multimodal representations. Curating such massive datasets involves substantial effort, including data collection, cleaning, filtering for quality, and sometimes human labeling to ensure proper alignment. Beyond data, computational costs are also very high. VLMs typically use large architectures, such as ViT for images and transformer-based encoders for text. Training them from scratch demands extensive GPU or TPU clusters running for weeks or even months. For example, models like CLIP (OpenAI) and ALIGN (Google) were trained on datasets that regular organizations cannot easily replicate due to hardware, storage, and energy costs. Moreover, achieving good generalization requires diverse and broad datasets, covering a wide range of visual and textual concepts, further increasing data acquisition challenges. Fine-tuning a pretrained VLM is more feasible for most organizations, but even that can be expensive if large-scale domain adaptation is needed.

因此，虽然从零开始开发虚拟语言模型（VLM）能够提供完全的控制权和潜在的创新空间，但其成本往往高得令人望而却步。如今，许多实际系统依赖于对开源预训练的VLM进行微调或适配，而不是训练全新的模型。构建多模态随机生成（RAG）系统是训练大型VLM的替代方案。如图2.3所示，在多模态RAG中，独立的检索器从外部源获取相关的文本、图像或混合模态数据，生成器则基于检索到的信息合成响应。这种方法利用现有的多模态嵌入和向量数据库，避免了大规模预训练的需要。它允许灵活地将文本、图像或两者作为下游任务（例如问答、图像描述或摘要）的上下文信息进行集成，使其成为一种……一种更高效、可扩展的多模态人工智能系统部署方法，无需承担端到端训练的高昂成本。

Therefore, while developing a VLM from scratch offers full control and potential innovation, it is often prohibitively expensive. Many practical systems today rely on fine-tuning or adapting open-source pretrained VLMs instead of training entirely new models. An alternative to training large VLMs from scratch is building multimodal RAG systems. In multimodal RAG, as shown in Figure 2.3, separate retrievers fetch relevant text, image, or mixed-modal data from external sources, and a generator synthesizes a response based on the retrieved information. This approach bypasses the need for massive pretraining by leveraging existing multimodal embeddings and vector databases. It allows flexible integration of text, images, or both as context for downstream tasks like QA, captioning, or summarization, making it a more efficient and scalable method for deploying multimodal AI systems without the heavy costs of end-to-end training.

让我们来了解一下多模态 RAG 系统，它是一种无需从头开始训练大型虚拟语言模型 (VLM) 即可高效构建多模态 AI 能力的方法。该系统能够智能地利用文本和图像数据来检索和生成答案。以下是该过程的详细步骤说明：

Let us understand the multimodal RAG system, an efficient way to build multimodal AI capabilities without training large VLMs from scratch. This system intelligently retrieves and generates answers by leveraging both text and image data. The following is a detailed step-by-step explanation of the process:

• 用户查询提交：用户通过提交查询来启动该过程。

  查询语句可以是：

  • 仅文本（例如，寻找带有良好摄像头的智能手机），
  • 仅图片（例如，上传智能手机照片），
  • 文字和图像的组合（例如，带有文字提示的照片，显示类似的模型）。

  该系统必须能够处理不同的模态并对其进行适当的解释。

• User query submission: A user initiates the process by submitting a query.

  The query can be:

  • Text-only (e.g., find smartphones with a good camera),
  • Image-only (e.g., uploading a smartphone photo),
  • A combination of text and image (e.g., a photo with the text prompt, show similar models).

  The system must handle different modalities and interpret them appropriately.

• 嵌入生成：查询内容（无论是文本、图像还是两者兼有）都会通过专门的嵌入模型进行处理，例如以下模型：
  • 文本嵌入模型：将文本输入转换为能够捕捉其语义含义的密集向量。
  • 图像嵌入模型：将图像转换为类似的高维向量空间表示。

  通过将两种模态编码到共享的嵌入空间中，该系统确保可以对文本和图像中的相似概念进行有意义的比较。

• Embedding generation: The query, whether text, image, or both, is processed through specialized embedding models, like the following:
  • Text embedding model: Converts the text input into a dense vector that captures its semantic meaning.
  • Image embedding model: Converts the image into a similar high-dimensional vector space representation.

  By encoding both modalities into a shared embedding space, the system ensures that similar concepts from text and images can be compared meaningfully.

• 知识库预处理：为检索做准备：
  • 文档被分成更小的、有意义的部分，称为块（例如，段落、章节）。
  • 如果图像有相应的文本元数据，也会对其进行处理并关联。
  • 文本块和图像都使用与查询不同的嵌入模型进行嵌入。但是请注意，相同的嵌入模型也可以在特定情况下使用，我们将在后续章节中讨论这一点。
  • 生成的嵌入向量存储在多模态向量数据库中。

  这一步骤确保所有知识资产（文本和图像）都可以通过矢量相似性进行搜索。

• Preprocessing of knowledge base: In preparation for retrieval:
  • Documents are broken into smaller, meaningful parts called chunks (e.g., paragraphs, sections).
  • Images are also prepared and associated with their corresponding textual metadata if available.
  • Both text chunks and images are embedded using the different embedding models used for the query. However, please take note that the same embedding models can also be used in specific contexts, and we will talk about this in the following chapters.
  • The resulting embeddings are stored inside a multimodal vector database.

  This step ensures that all knowledge assets—text and images—are searchable through vector similarity.

• 向量数据库搜索：查询嵌入后：
  • 该系统针对多模态向量数据库执行向量搜索。
  • 它会找到与用户查询语义最相似的文档、段落或图像。
  • 检索是在嵌入空间中进行的，这使得匹配可以超越精确的关键词或像素相似性，实现更灵活的匹配。

  此搜索步骤确保获取最相关的知识片段，无论其形式如何。

• Vector database search: Once the query is embedded:
  • The system performs a vector search against the multimodal vector database.
  • It finds the most semantically similar documents, paragraphs, or images to the user's query.
  • The retrieval happens in embedding space, which allows for flexible matching beyond exact keyword or pixel similarity.

  This search step ensures that the most relevant knowledge pieces, irrespective of modality, are fetched.

• 检索结果汇总：
  • 矢量搜索结果包括文本块、图像或两者兼有。
  • 从检索到的上下文中得到的这些结果将有助于最终响应的生成。
  • 检索到的内容为生成模型提供了事实基础，提高了模型的准确性和相关性。
• Retrieved results consolidation:
  • The vector search results include text chunks, images, or both.
  • These results from the retrieved context will assist the final response generation.
  • Retrieved content provides factual grounding for the generative model, improving its accuracy and relevance.
• LLM 的响应生成：
  • 然后将检索到的多模态内容输入到 LLM 中。
  • LLM 使用检索到的知识作为上下文，综合出最终答案或输出。
  • 它可以根据查询内容，结合文字解释、描述图像或生成创意输出。

  这种设计确保模型不会凭空产生答案，而是将答案建立在实际检索到的知识之上。

• Response generation by LLM:
  • The retrieved multimodal content is then fed into a LLM.
  • The LLM synthesizes the final answer or output using the retrieved knowledge as context.
  • It may combine text explanations, describe images, or generate creative outputs depending on the query.

  This design ensures that the model does not hallucinate answers but grounds them in actual retrieved knowledge.

• 返回结果：
  • 最终结果将返回给用户。
  • 输出结果可能包括基于文本的答案、图像引用或基于检索信息的多模态摘要。

  因此，用户可以通过检索和增强高效地获得高质量的响应。

• Returning the result:
  • The final result is returned to the user.
  • The output could include text-based answers, references to images, or multimodal summaries based on the retrieved information.

  Thus, users receive high-quality responses generated efficiently through retrieval and augmentation.

这种多模态 RAG 架构高效地融合了文本和图像检索与生成能力。它无需进行大规模的虚拟模型预训练，降低了计算成本，并实现了多模态系统的可扩展部署。通过分离检索和生成过程，组织可以利用现有的嵌入模型和逻辑层模型构建强大的 AI 解决方案，使其成为实际多模态 AI 应用的理想选择。

This multimodal RAG architecture efficiently merges text and image retrieval with generative capabilities. It bypasses the need for massive VLM pretraining, reduces computational costs, and enables scalable deployment of multimodal systems. By separating retrieval and generation, organizations can build powerful AI solutions with existing embedding models and LLMs, making it an attractive option for real-world multimodal AI applications.

多模态向量嵌入

Multimodal vector embedding

现在您应该明白，在全人类人工智能（GenAI）时代，跨多种模态（文本、图像、音频、视频和结构化数据）处理数据的能力不再是锦上添花，而是必不可少。多模态红绿灯（RAG）系统正处于这一演进的前沿，它通过从各种数据源检索相关信息来增强逻辑逻辑模型（LLM），从而实现更具上下文关联性、信息量更大、更接近人类的响应。然而，此类系统的有效性很大程度上取决于其底层向量表示，特别是生成多模态向量嵌入的能力，该向量嵌入能够将不同格式的信息统一到一个可比较的、语义丰富的空间中。

Now you know that in the era of GenAI, the ability to work across multiple modalities, text, images, audio, video, and structured data, is no longer a luxury but a necessity. Multimodal RAG systems are at the forefront of this evolution, enabling more context-rich, informative, and human-like responses by augmenting LLMs with relevant information retrieved from diverse data sources. However, the effectiveness of such systems is heavily dependent on their underlying vector representations, specifically, the ability to generate multimodal vector embeddings that unify information across formats in a comparable, semantically rich space.

多模态向量嵌入至关重要，因为它们构成了 RAG 流程中相似性搜索的核心。对于仅限于文档、网页或文本知识库的应用，标准的纯文本 RAG 系统或许就足够了。然而，现实世界的信息通常是多模态的。例如，用户手册包含文本和图表；产品规格包含表格数据和带注释的图像；客户支持互动可能涉及语音转录和屏幕截图。如果系统无法同时理解和检索这些异构格式中的相关信息，就会错过关键信号，导致生成质量欠佳。

Multimodal vector embeddings are essential because they form the backbone of similarity search in a RAG pipeline. A standard text-only RAG system may suffice for applications limited to documents, webpages, or textual knowledge bases. However, real-world information is often multimodal. For example, user manuals contain both text and diagrams; product specifications include tabular data and annotated images; customer support interactions may involve voice transcripts and screenshots. A system that cannot simultaneously understand and retrieve relevant information from these heterogeneous formats will miss critical signals, leading to suboptimal generation quality.

为了实现跨模态检索，每段内容（无论是图像、段落还是音频片段）都必须嵌入到向量空间中。然而，与所有嵌入都源自同一编码器并存在于统一潜在空间的单模态系统不同，多模态系统需要更复杂的设计。如图2.2所示，通常使用不同的编码器（例如，图像使用 CLIP，文本使用Sentence Transformer ，音频使用 Whisper）来生成特定模态的嵌入。这些嵌入必须映射到共享的潜在空间，或者通过索引策略进行链接，以便高效地计算跨模态的相似度。

To enable cross-modal retrieval, each piece of content, whether it is an image, paragraph, or audio clip, must be embedded into a vector space. However, unlike unimodal systems, where all embeddings are derived from the same encoder and live in a uniform latent space, multimodal systems require a more sophisticated design. As explained in Figure 2.2, separate encoders (e.g., CLIP for images, Sentence Transformers for text, and Whisper for audio) are often used to generate modality-specific embeddings. These embeddings must then either be mapped into a shared latent space or linked via indexing strategies that allow for efficient similarity computation across modalities.

例如，假设用户上传了一张笔记本电脑侧面轮廓图，并询问“给我展示带有类似这种接口的笔记本电脑” 。单模态 RAG 系统无法识别这张图片。相比之下，采用联合向量嵌入的多模态 RAG 系统可以将这张图片与数据库中存储的类似笔记本电脑接口图进行匹配，并检索相应的产品规格和评价。这种检索之所以可行，是因为视觉信息和文本信息都被表示为共享或对齐空间中的向量，从而保留了语义信息。

For example, consider a user asking, show me laptops with ports like this, while uploading an image of a laptop side profile. A unimodal RAG system would fail to interpret the image. In contrast, a multimodal RAG system with joint vector embeddings can match the image to similar laptop port diagrams stored in the database and retrieve corresponding product specifications and reviews. This retrieval is only possible because the visual and textual information are both represented as vectors in a shared or aligned space that preserves semantic meaning.

多模态向量嵌入也增强了查询构建的灵活性。用户可以输入图像、文本，甚至二者的组合，系统可以将其与相关的文档、图表或知识库进行匹配。这使得系统更加直观和包容，能够跨越语言障碍，并方便那些可能没有精确关键词但拥有视觉或听觉线索的用户。

Multimodal vector embeddings also enhance the flexibility of query formulation. Users can input images, text, or even a combination of both, and the system can match them against relevant documents, diagrams, or knowledge chunks. This makes the system more intuitive and inclusive, bridging language barriers and accommodating users who may not have the precise keywords but possess visual or auditory cues.

此外，在专为医疗保健、法律或制造业等高风险领域设计的 RAG 系统中，使用多模态嵌入可以确保答案生成拥有更全面的证据基础。它通过将答案生成与真实的多模态数据样本挂钩，而不是仅仅依赖于先验模型知识，从而降低了出现幻觉的风险。

Furthermore, in RAG systems designed for high-stakes domains like healthcare, legal, or manufacturing, the use of multimodal embeddings ensures a more comprehensive evidence base for answer generation. It reduces the risk of hallucinations by anchoring the generation to real, multimodal data artifacts rather than relying purely on prior model knowledge.

多模态向量数据库

Multimodal vector database

一旦生成了多模态向量嵌入（用于在共享语义空间中表示文本、图像或两者），就必须对其进行高效存储和检索，以支持实时人工智能应用。这时，多模态向量数据库就显得至关重要。它提供了一个结构化的高性能存储系统，针对不同模态嵌入的相似性搜索进行了优化。通过将这些嵌入与元数据（例如语言、时间戳）一起组织起来，向量数据库能够实现快速、过滤的近似最近邻（ANN ）检索。从嵌入到向量数据库的这种转变对于构建可扩展的跨模态系统（例如多模态红绿灯算法、推荐引擎和语义搜索平台）至关重要。

Once multimodal vector embeddings are generated, representing text, images, or both in a shared semantic space, they must be efficiently stored and retrieved to support real-time AI applications. This is where a multimodal vector database becomes essential. It provides a structured, high-performance storage system optimized for similar search across embeddings from different modalities. By organizing these embeddings alongside metadata (e.g., language, timestamp), the vector database enables fast, filtered approximate nearest neighbor (ANN) retrieval. This transition from embeddings to a vector database is crucial for powering scalable, cross-modal systems such as multimodal RAG, recommendation engines, and semantic search platforms.

例如：

Examples:

• 只要满足以下条件，Qdrant、Weaviate、Pinecone、Milvus 和 Chroma 都可以用作多模态矢量数据库：
  • 您可以将文本和图像嵌入存储在同一个集合中，或者逻辑上分组的集合中。
  • 您可以将来自不同模态的嵌入规范化到同一维度（通常是必要的）。
  • 您可以使用适当的元数据标签（例如，“type”:“text”或“type”:“image” ）来控制检索行为。
• Qdrant, Weaviate, Pinecone, Milvus, and Chroma can all be used as multimodal vector databases, provided:
  • You store both text and image embeddings in the same or logically grouped collections.
  • You normalize embeddings from different modalities into the same dimension (often necessary).
  • You use appropriate metadata tags (e.g., "type": "text" or "type": "image") to control retrieval behavior.

在使用多模态向量数据库时，例如使用 Qdrant 作为存储和检索高维多模态嵌入的向量数据库，您必须考虑一些关键的设计选择。

You have to touch on some critical design choices when using a multimodal vector database, let us say Qdrant as a vector database for storing and retrieving high-dimensional multimodal embeddings.

让我们结合流行的矢量数据库 Qdrant，具体了解几个关键概念。虽然大多数矢量数据库的运行原理类似，但逐一详细介绍每个数据库超出了本章和本书的范围。

Let us understand a few key concepts specifically in the context of Qdrant, a popular vector database. While most vector databases operate on similar principles, detailing each one individually is beyond the scope of this chapter and book.

收藏

Collections

在 Qdrant 中，集合是基本的组织单元。它本质上是一组带有标签的数据点，这些数据点共享一个共同的结构。集合中的每个点都与一个固定大小的向量相关联，并使用特定的相似度度量（例如，余弦相似度、点积、欧氏距离）进行比较。同一集合中的所有向量必须遵循统一的维度和距离函数。Qdrant 还允许在单个点中使用不同的名称存储多个向量，称为命名向量，每个命名向量可以单独遵循不同的度量和维度设置。

A collection is the fundamental organizational unit in Qdrant. It is essentially a labeled group of data points that share a common structure. Each point in a collection is associated with a vector of a fixed size and is compared using a specific similarity metric (e.g., cosine, dot product, Euclidean). All vectors in the same collection must adhere to this uniform dimensionality and distance function. Qdrant also allows multiple vectors to be stored under different names within a single point called named vectors, which can individually follow different metric and dimension settings.

点和点 ID

Points and point IDs

在 Qdrant 中，点是指集合中的一个独立条目。它包含以下内容：

In Qdrant, a point is an individual entry within a collection. It comprises of the following:

• 唯一标识符（点 ID）。
• A unique identifier (point ID).
• 一个或多个向量嵌入。
• One or more vector embeddings.
• 可选元数据，称为有效载荷。
• Optional metadata known as payload.

这些点是用户使用向量相似度进行搜索的基本单元。点 ID 用于检索、更新或删除特定记录。所有与点相关的操作，包括插入或更新，都会首先被记录，以确保数据的持久性和恢复能力，即使在断电的情况下也能正常运行。

These points are the basic units that users search against using vector similarity. The point ID is used to retrieve, update, or delete specific records. All point-related operations, including insertions or updates, are first logged to ensure durability and recovery, even in the event of power failure.

向量

Vectors

向量（也称为嵌入）表示各种数据类型（例如图像、文本或音频）的编码数值形式。这些向量使得在高维空间中比较不同的数据对象成为可能。两个向量在这个空间中越接近，它们所代表的原始对象就越相似。为了生成这些嵌入，通常使用神经网络，该网络经过训练以学习有意义的模式，通常基于对已标记或弱标记数据的对比学习。向量是相似性搜索的基石，并被应用于聚类、排序和检索任务中。

Vectors (also known as embeddings) represent the encoded numerical form of various data types, such as images, text, or audio. These vectors enable the comparison of different data objects in high-dimensional space. The closer two vectors are in this space, the more similar their original objects are considered to be. To generate these embeddings, one typically uses a neural network trained to learn meaningful patterns, often based on contrastive learning from labeled or weakly labeled data. Vectors are the cornerstone of similarity search and are used in clustering, ranking, and retrieval tasks.

有效载荷

Payload

有效载荷是指与每个向量一起存储的附加元数据。这些元数据非常灵活，可以采用任何 JSON 兼容的结构。它可以描述语言、时间戳、用户信息、类别或任何特定领域的标签等属性。有效载荷使 Qdrant 能够执行过滤搜索，允许用户将相似性搜索限制在具有特定属性的向量上。 元数据属性。例如，仅检索英文文档或按日期筛选。

The payload refers to additional metadata stored alongside each vector. This metadata is flexible and can take any JSON-compatible structure. It can describe attributes like language, timestamp, user information, category, or any domain-specific tags. Payloads allow Qdrant to perform filtered searches, letting users restrict similarity searches to vectors with certain metadata properties. For example, retrieving only English-language documents or filtering by date.

存储和矢量存储

Storage and vector store

Qdrant 将数据组织成各个集合内的分段。每个分段都维护着自己的一组向量、有效载荷和索引。分段针对不同的使用场景进行了优化，例如：

Qdrant organizes its data into segments within each collection. Each segment maintains its own set of vectors, payloads, and indexes. Segments are optimized for different use cases, like the following:

• 可追加段支持快速插入、更新和删除。
• Appendable segments support fast inserts, updates, and deletions.
• 不可追加段针对静态数据或读取密集型数据进行了优化。
• Non-appendable segments are optimized for static or read-heavy data.

Qdrant 支持两种存储模型，分别是：

Qdrant supports two storage models, which are as follows:

• 内存存储：将所有向量数据保存在 RAM 中，以实现最佳性能，磁盘仅用于持久化存储。
• In-memory storage: Keeps all vector data in RAM for maximum performance, with disk used only for persistence.
• 内存映射存储：将磁盘文件链接到虚拟内存，通过利用操作系统的页面缓存，在速度和内存使用之间取得平衡。
• Memory-mapped storage: Links disk files to virtual memory, offering a balance between speed and memory usage by leveraging the operating system’s page cache.

这种架构确保可以根据应用程序需求调整性能和成本。

This architecture ensures that performance and cost can be tuned based on application requirements.

索引

Indexing

在实际的多模态 GenAI 系统中，高效的数据管理和检索对于在图像和文本模态上提供快速、准确的响应至关重要。高性能矢量数据库 Qdrant 通过结合矢量索引和有效载荷过滤，实现了这一目标，确保语义相似性和结构化元数据约束都能得到无缝处理。Qdrant 利用集合、点级元数据（有效载荷）以及来自 CLIP 或 BLIP 等模型的高维嵌入，支持混合搜索——根据语义和产品类别或颜色等过滤器检索相关项。这些索引策略，包括分层可导航小世界( HNSW ) 索引和有效载荷索引，确保 GenAI 应用在保持低延迟性能的同时，实现可靠的扩展。Qdrant 同时支持矢量索引和有效载荷（过滤器）索引，从而实现高效的混合搜索：

In a real-world multimodal GenAI system, efficient data management and retrieval are essential for delivering fast, accurate responses across image and text modalities. Qdrant, a high-performance vector database, enables this by combining vector indexing and payload filtering, ensuring both semantic similarity and structured metadata constraints are handled seamlessly. By leveraging collections, point-level metadata (payloads), and high-dimensional embeddings from models like CLIP or BLIP, Qdrant facilitates hybrid search—retrieving relevant items based on meaning and filters like product category or color. These indexing strategies, including hierarchical navigable small world (HNSW) and payload indexes, ensure GenAI applications scale reliably while maintaining low-latency performance. Qdrant supports both vector indexing and payload (filter) indexing, allowing efficient hybrid search:

• 向量索引（例如 HNSW）通过将向量组织成图结构来加速相似性搜索，从而减少搜索空间。
• Vector indexes (e.g., HNSW) accelerate similarity searches by organizing vectors into graph structures that reduce the search space.
• 有效载荷索引的功能与传统数据库中的索引类似。它们允许基于元数据（例如，语言或类别字段）进行快速筛选。
• Payload indexes function similarly to indexes in traditional databases. They allow for fast filtering based on metadata (e.g., language or category fields).

索引虽然能提高速度和准确性，但也会增加内存和处理成本。用户可以根据预期的查询模式和基数，选择性地配置要建立索引的字段。索引参数在集合级别定义，但索引在段中的实际存在与否取决于优化规则和数据分布。

While indexing improves speed and accuracy, it incurs additional memory and processing costs. Users can selectively configure which fields should be indexed based on their expected query patterns and cardinality. Index parameters are defined at the collection level, but the actual index presence in segments depends on optimization rules and data distribution.

让我们把您在 Qdrant 中学到的所有概念直接整合到构建多模态 GenAI 系统的背景下。

Let us integrate all the Qdrant concepts you learned about directly into the context of building a Multimodal GenAI system.

在实际的多模态GenAI系统中，管理和检索跨模态（例如文本和图像）的数据不仅仅是嵌入向量；更重要的是高效地大规模组织、过滤和检索这些数据。正因如此，诸如集合、点、向量、有效载荷和索引等概念（如Qdrant等向量数据库中所实现的）才显得至关重要。

In a practical multimodal GenAI system, managing and retrieving data across modalities, like text and images, is not just about embedding vectors; it is about organizing, filtering, and retrieving them efficiently at scale. This is where concepts like collections, points, vectors, payloads, and indexes, as implemented in vector databases such as Qdrant, become critically important.

此类系统的核心是向量嵌入过程。对于每种输入数据类型，例如产品图像或其描述，神经网络模型（例如 CLIP 或 BLIP）会将输入转换为高维向量。这些向量捕捉语义信息，因此“红色跑车”这样的描述和红色跑车的图像会生成在向量空间中彼此接近的嵌入向量。然后，这些嵌入向量被分组到集合中，每个集合代表一个逻辑数据集片段。例如，一个集合可以存储所有与零售产品数据相关的向量，图像和文本作为命名向量存储在每个点下。

At the core of such a system is the vector embedding process. For each input data type, such as a product image or its description, a neural network model (e.g., CLIP or BLIP) converts the input into a high-dimensional vector. These vectors capture semantic meaning, so a caption like a red sports car and an image of a red sports car will generate embeddings that lie close to each other in the vector space. These embeddings are then grouped into collections, each representing a logical dataset segment. For example, a single collection may store all vectors related to retail product data, with images and text stored as named vectors under each point.

该集合中的每个点代表一个单独的项目，例如一个产品实例，并被分配一个唯一的点 ID。除了向量之外，点还可以包含有效载荷，其中存储着有用的元数据，例如语言、时间戳、产品类别，甚至是原始文件来源。在多模态 GenAI 设置中，当我们在检索过程中需要按模态、时间范围或其他条件筛选结果时，此有效载荷就显得至关重要。

Each point within this collection represents an individual item, say, a product instance, and is assigned a unique point ID. Alongside the vector(s), a point can include a payload, which stores useful metadata such as language, timestamp, product category, or even the original file source. In a multimodal GenAI setup, this payload becomes crucial when we want to filter results by modality, time range, or other criteria during retrieval.

当用户输入查询时，例如上传一张产品图片并附带类似“显示蓝色款的类似型号”这样的文字请求，系统需要执行混合搜索。这意味着不仅要基于向量相似度检索结果，还要使用有效负载中定义的约束条件（例如，color = "blue" ）。为了实现这一点，Qdrant 支持有效负载索引，它允许对结构化元数据字段进行快速筛选，类似于传统关系数据库中的索引。

When a user inputs a query, perhaps a product photo with a textual request like show similar models available in blue, the system needs to perform a hybrid search. This means retrieving results not only based on vector similarity but also using constraints defined in the payload (e.g., color = "blue"). To enable this, Qdrant supports payload indexing, which allows fast filtering across structured metadata fields, much like indexes in traditional relational databases.

在后台，数据集被划分为多个段，每个段都有自己的存储和索引配置。根据性能需求，这些段可能使用内存存储以实现最高速度，也可能使用内存映射存储来优化 RAM 使用，同时仍能通过操作系统级页面缓存实现快速访问。对于服务数百万用户的实际 GenAI 应用而言，这种存储分段方式可确保可扩展性和容错性。

Behind the scenes, the collection is divided into segments, each with its own storage and indexing configuration. Depending on performance requirements, these segments may use in-memory storage for maximum speed or memory-mapped storage to optimize RAM usage while still enabling fast access via the OS-level page cache. For a real-world GenAI application that serves millions of users, segmenting storage this way ensures both scalability and fault tolerance.

最后，为了加速向量检索，Qdrant 支持高性能向量索引（例如 HNSW），使系统能够在无需暴力比较的情况下，快速逼近高维空间中的最近邻。结合有效载荷滤波器，这种索引策略能够实现精确控制的 ANN 检索，这对于实时多模态系统至关重要。

Finally, to accelerate vector retrieval, Qdrant supports high-performance vector indexes (e.g., HNSW) that allow the system to quickly approximate the nearest neighbors in high-dimensional space without brute-force comparison. Combined with payload filters, this indexing strategy enables ANN retrieval with precise control, which is vital for real-time multimodal systems.

实施方案比较

Implementation comparisons

在存储和搜索高维多模态嵌入时，通常有两种策略：使用带有过滤器的单个集合，以及创建多个带有局部索引的集合。当嵌入频繁更新时（例如在生产流程中每天更新），以及当查询需要细粒度控制时，这些设计决策尤为重要。

Two common strategies emerge when storing and searching high-dimensional multimodal embeddings: using a single collection with filters vs. creating multiple collections with localized indexing. These design decisions are especially important when embeddings are updated frequently, such as every day in a production pipeline, and when queries require fine-grained control.

单个集合，按有效载荷分区

Single collection, partitioned via payload

在这种方法中，所有向量嵌入，无论源自图像、文本还是多模态文档，都存储在一个统一的集合中。数据类型（例如日期或语言）的区分通过有效负载元数据来实现。例如，每个数据点可能带有类似{"date": "2025-05-09", "language": "en"} 的标签，这些标签在查询执行期间用作过滤器。

In this approach, all vector embeddings, whether derived from images, text, or multimodal documents, are stored in a single, unified collection. The differentiation between data types, such as dates or languages, is handled via payload metadata. For example, each point might carry tags like {"date": "2025-05-09", "language": "en"}, which are then used as filters during query execution.

这种设置简单且可扩展。只需维护一个数据集，所有嵌入都可以在同一个向量空间中搜索。在操作上，它成本效益高，并且易于与下游系统集成。然而，由于没有构建跨子集（例如，按日期或语言）的全局索引，人工神经网络（ANN）的检索准确率显著降低，与精确的K近邻（KNN）搜索相比，匹配率仅为50%左右。

This setup is simple and scalable. There is only one collection to maintain, and all embeddings are searchable in a single vector space. Operationally, it is cost-efficient and easy to integrate with downstream systems. However, because no global index is built across subsets (e.g., by date or language), the ANN retrieval accuracy is significantly lower, dropping to around a 50% match rate compared to exact KNN searches.

仅通过有效载荷过滤嵌入向量而不进行全局索引会造成效率低下，尤其是在数据集增长或数据在时间和类别上出现偏差时。例如，如果某个日期包含的数据量异常多，或者某种特定语言的数据量占主导地位，则由于过滤后的子集中向量分布不均，人工神经网络 (ANN) 搜索可能会降低精度。下图展示了一个通过有效载荷方法进行分割的数据集：

Filtering embeddings purely through payload without global indexing introduces inefficiencies, especially when the dataset grows or becomes skewed across time and classes. For example, if one date contains disproportionately more data or a specific language dominates, ANN search may lose precision due to uneven vector distribution across the filtered subsets. The following figure depicts a single collection, portioned via a payload approach:

使用案例：此方法最适合那些优先考虑易于维护和成本控制而非完美检索准确性的环境，例如非关键检索任务或早期原型。

Use case: This method is best for environments where ease of maintenance and cost control are prioritized over perfect retrieval accuracy, such as in non-critical retrieval tasks or early-stage prototypes.

具有全局索引的多个集合

Multiple collections with global indexing

第二种策略选择按日期划分集合，每天创建一个集合（例如，embeddings_2025_05_08、embeddings_2025_05_09 ）。在每个集合中，显式构建一个全局向量索引（例如，HNSW），以实现高度优化的人工神经网络检索。然后，可以使用有效载荷过滤器按语言进一步划分每个集合。

The second strategy opts for separating collections by date, creating one collection per day (e.g., embeddings_2025_05_08, embeddings_2025_05_09). Within each collection, a global vector index (e.g., HNSW) is explicitly built to enable highly optimized ANN retrieval. Each collection can then be partitioned further by language using payload filters.

这种方法显著提高了基于人工神经网络（ANN）的搜索精度——与精确KNN相比，匹配率高达98%——因为每个集合都受益于局部索引和更均匀的嵌入分布。通过将搜索空间缩小到单个日期并仅在该时间段内进行过滤，该系统避免了大型全局集合中常见的向量聚类稀释问题。

This approach results in significantly higher precision during ANN-based searches—up to 98% match rate compared to exact KNN—because each collection benefits from localized indexing and a more homogeneous embedding distribution. By narrowing the search space to a single date and filtering only within that segment, the system avoids the dilution of vector clusters that occurs in large, global collections.

然而，这种模式是有代价的。维护多个集合会增加操作复杂性，系统必须管理每个新集合的索引成本。此外，随着时间的推移（例如，按小时或按用户），扩展到大量集合可能会导致资源效率低下和存储开销。

However, this model comes at a cost. Maintaining multiple collections increases operational complexity, and the system must manage the indexing cost for each new collection. Additionally, scaling to many collections over time (e.g., per hour or per user) may lead to resource inefficiency and storage overhead.

使用场景：当需要高精度推荐或精确语义搜索时，例如在产品推荐引擎、个性化助手或关键分析管道中，此模型是理想之选。

Use case: This model is ideal when high-accuracy recommendations or precise semantic search are required, such as in product recommendation engines, personalized assistants, or critical analytics pipelines.

下图展示了采用全局索引方法的多个集合：

The following figure depicts multiple collections with a global indexing approach:

在探讨了如何使用向量数据库组织、存储和检索多模态向量嵌入之后，我们现在将重点转向如何将这些功能应用于端到端人工智能系统。向量数据库是高效存储和搜索的基础。在各种模态下，基于此基础设施的架构选择，特别是使用多模态 GenAI 系统还是虚拟线性模型 (VLM)，会对性能、可扩展性和应用契合度产生显著影响。在接下来的章节中，我们将探讨这两种方法之间的根本区别，并分析各自的适用场景。

Having explored how multimodal vector embeddings are organized, stored, and retrieved using vector databases, we now shift our focus to how these capabilities are applied in end-to-end AI systems. While vector databases serve as the backbone for efficient storage and search across modalities, the architectural choices made on top of this infrastructure, particularly whether to use a multimodal GenAI system or a VLM, can significantly influence performance, scalability, and application fit. In the following section, we examine the fundamental differences between these two approaches and look at when each should be used.

多模态生成式人工智能系统与虚拟语言模型

Multimodal generative AI systems vs. VLMs

随着人工智能的演进，对能够理解和生成多种数据模态（例如文本、图像、音频或结构化数据）的系统的需求显著增长。两种方法已脱颖而出，成为这一发展趋势的前沿：虚拟语言模型（VLM）和更广泛的多模态生成人工智能（GenAI）系统。虽然这两个术语有时可以互换使用，但它们用途不同，架构原则也不同。本节将阐明它们的区别，并就何时最适合应用每种方法提供指导。

As AI evolves, the demand for systems that can understand and generate across multiple data modalities, such as text, images, audio, or structured data, has grown significantly. Two approaches have emerged at the forefront of this advancement: VLMs and broader multimodal GenAI systems. While the terms are sometimes used interchangeably, they serve distinct purposes and operate under different architectural principles. This section clarifies their differences and offers guidance on when each is best applied.

视觉语言模型

Vision-language models

视觉语言模型（VLM）是多模态人工智能系统的一个子集，专门用于整合视觉和文本模态。这些模型经过训练，能够理解图像特征并与语言特征相匹配，从而实现图像描述、视觉问答、图像文本检索和跨模态推理等任务。

VLMs are a subset of multimodal AI systems that specifically integrate visual and textual modalities. These models are trained to understand and align image features with language features, enabling tasks such as image captioning, VQA, image-text retrieval, and cross-modal reasoning.

视觉学习模型（VLM）通常采用融合两种独立神经编码器嵌入的架构：视觉编码器（例如 ViT、ResNet）和文本编码器（例如 BERT、RoBERTa 或 GPT）。融合后的表示使模型能够跨模态进行推理。一些模型使用交叉注意力机制，使图像标记能够关注文本标记，反之亦然；而另一些模型则使用对比学习（例如 CLIP、ALIGN）将图像和文本映射到共享的潜在空间以进行检索。

VLMs are typically built using architectures that fuse embeddings from two separate neural encoders: a vision encoder (e.g., ViT, ResNet) and a text encoder (e.g., BERT, RoBERTa, or GPT). The fused representation allows the model to reason across both modalities. Some models use cross-attention mechanisms to allow image tokens to attend to text tokens, and vice versa, while others use contrastive learning (e.g., CLIP, ALIGN) to map images and texts into a shared latent space for retrieval.

以下是VLM的示例：

The following are examples of VLMs:

• 夹子
• CLIP
• BLIP 和 BLIP-2
• BLIP and BLIP-2
• UNITER、LXMERT、VisualBERT
• UNITER, LXMERT, VisualBERT
• 用于少样本多模态推理的 Flamingo 和 Kosmos-1
• Flamingo and Kosmos-1 for few-shot multimodal reasoning

VLM 通常使用自监督或半监督学习目标，在大型图像-文本数据集上进行预训练，并且可以针对需要视觉语言对齐的下游任务进行微调。

VLMs are often pretrained on large image-text datasets, using self-supervised or semi-supervised learning objectives, and can be fine-tuned on downstream tasks requiring vision-language alignment.

多模态生成式人工智能系统

Multimodal generative AI systems

相比之下，多模态GenAI系统旨在跨多种模态运行，这些模态通常是任意的，而不仅限于视觉和语言。这些系统结合了检索、推理和生成等组件，通常通过模块化架构进行协调。

In contrast, multimodal GenAI systems are designed to operate across multiple and often arbitrary modalities, not limited to vision and language. These systems combine components for retrieval, reasoning, and generation, often orchestrated through modular architectures.

一个关键区别在于多模态 GenAI 系统中常用的检索增强架构。这些系统并非仅仅依赖单个预训练模型，而是：

A key difference is the retrieval-augmented architecture often used in multimodal GenAI systems. Instead of relying solely on a single pretrained model, these systems:

• 从外部存储中检索相关的文本、图像或结构化数据。
• Retrieve relevant text, image, or structured data from external stores.
• 对这些信息进行编码和融合。
• Encode and fuse that information.
• 使用语言模型、图像生成器或其他特定模态生成器生成输出。
• Generate outputs using a language model, image generator, or other modality-specific generator.

多模态GenAI系统可以将虚拟线性模型（VLM）作为子组件集成，但并不局限于此。它们通常支持以下功能：

Multimodal GenAI systems can incorporate VLMs as subcomponents but are not limited to them. They often support the following:

• 图片+文字→结构化报告
• Image + text → Structured report
• 音频+文本→摘要
• Audio + text → Summary
• 文字→图片 + 标题
• Text → image + Caption
• 图片→文本 + SQL 查询
• Image → text + SQL query
• PDF + 图表→自然语言解释
• PDF + diagram → Natural language explanation

这些系统通常基于流水线，结合不同的模型和检索层来执行任务。RAG（资源获取、请求生成）、编排层（例如 LangChain 或 LangGraph）以及工具的使用（通过代理）都很常见。

These systems are typically pipeline-based, combining different models and retrieval layers to perform a task. RAG, orchestration layers (like LangChain or LangGraph), and tool use (via agents) are common.

让我们快速了解一下建筑上的差异：

Let us look at the architectural differences at a glance:

表 2.1：VLM 与多模态 GenAI 系统的比较

Table 2.1: Comparison of VLMs vs. multimodal GenAI systems

使用视觉语言模型

Using vision-language models

在以下情况下应使用 VLM：

You should use VLMs in the following cases:

• 您的任务与视觉和文本对齐密切相关，例如：
  • 根据文本提示进行图像分类
  • 字幕生成
  • VQA
  • 图像-文本或文本-图像检索
• Your task is tightly coupled with visual and textual alignment, such as:
  • Image classification with text prompts
  • Caption generation
  • VQA
  • Image-text or text-image retrieval
• 你需要端到端的性能，低延迟推理，并且不需要外部检索逻辑。
• You need end-to-end performance with low-latency inference and no external retrieval logic.
• 您目前使用的模型有限，并且更喜欢单模型设置而不是流水线编排。
• You are working with limited modalities and prefer a single-model setup over pipeline orchestration.
• 你需要使用在大规模数据集上预训练的跨模态理解模型，并且可以针对特定任务进行微调。
• You require models that have been pretrained on large-scale datasets for cross-modal understanding and can be fine-tuned for specialized tasks.
• 您的环境更倾向于模型推理，而不是动态组合，例如移动应用程序或边缘设备。
• Your environment favors model inference over dynamic composition, such as mobile applications or edge devices.

简而言之，VLM 非常适合受控的视觉语言任务，这些任务可以从单个模型中的深度跨模态表征学习中受益。

In short, VLMs are ideal for controlled, vision-language tasks that benefit from deep cross-modal representation learning in a single-model.

使用多模态生成式人工智能系统

Using multimodal generative AI systems

在以下情况下使用多模态GenAI系统：

Use multimodal GenAI systems when:

• 除了视觉和文本之外，你还需要结合多种其他模态，例如：
  • 文本+图像+表格（例如，文档解析）
  • 音频+文本（例如，会议摘要）
  • PDF + 图表 + 问题（例如，科学论文分析）
• You need to combine multiple modalities beyond vision and text, such as:
  • Text + image + tables (e.g., document parsing)
  • Audio + text (e.g., meeting summarization)
  • PDF + chart + question (e.g., scientific paper analysis)
• 你需要检索增强推理：
  • 搜索矢量数据库
  • 检索相关文档或图像
  • 使用外部电源接地输出
• You require retrieval-augmented reasoning:
  • Search a vector database
  • Retrieve related documents or images
  • Ground the output using external sources
• 您的任务需要使用工具、决策逻辑或基于代理的编排：
  • 数据库查询（SQL 生成）
  • 调用 API
  • 多步骤推理（例如，问题分解）
• Your task demands tool use, decision logic, or agent-based orchestration:
  • Querying databases (SQL generation)
  • Invoking APIs
  • Multi-step reasoning (e.g., question decomposition)
• 您需要的是一种灵活且模块化的架构：
  • 可更换的取回器
  • 自定义嵌入模型
  • 可替换的大型语言模型（OpenAI、Ollama、Claude 等）
• You want a flexible and modular architecture:
  • Replaceable retrievers
  • Custom embedding models
  • Swappable large language models (OpenAI, Ollama, Claude, etc.)
• 你需要一个定期更新的动态知识库：
  • 嵌入更新
  • 多语言检索
  • 使用元数据进行过滤搜索（例如，通过 Qdrant 或 Weaviate）
• You need a dynamic knowledge base that updates regularly:
  • Embedding updates
  • Multilingual retrieval
  • Filtered search using metadata (e.g., via Qdrant or Weaviate)

从本质上讲，多模态 GenAI 适用于需要高度适应性、实时数据集成和 AI 组件复杂编排的企业级、多用途应用。

In essence, multimodal GenAI is suited for enterprise-grade, multi-purpose applications that demand high adaptability, live data integration, and sophisticated orchestration of AI components.

实际案例比较

Real-world example comparison

让我们根据一份包含文字和图片的产品手册来回答一个问题。

Let us take the task of answering a question based on a product manual that includes both text and figures.

• VLM 可以给图像添加说明文字，或者回答一些基本的视觉问题，例如开关是什么颜色的？
• A VLM might be able to caption the image or answer basic visual questions, such as what color is the switch?
• 然而，多模态GenAI系统可以：
  • 解析PDF文件
  • 提取图表
  • 对嵌入式图像使用OCR技术
  • 获取相关产品规格
  • 使用语言模型生成完整的响应，例如：要激活省电模式，请按下图 2.3所示的绿色开关。该开关控制辅助电路。
• A multimodal GenAI system, however, could:
  • Parse the PDF
  • Extract diagrams
  • Use OCR on embedded images
  • Retrieve relevant product specs
  • Use a language model to generate a full response like: to activate the power-saving mode, press the green switch shown in Figure 2.3. The switch controls the secondary circuit.

这种涵盖文本、图像、布局和逻辑的端到端能力是多模态 GenAI 的标志。

This end-to-end capability across text, image, layout, and logic is the hallmark of multimodal GenAI.

基于输出的多模态系统分类

Output-based classification of multimodal systems

多模态全人类人工智能系统不仅能够处理多种类型的输入，例如文本、图像、音频或结构化数据，而且还能够生成多种输出。随着各组织在电子商务、医疗保健、软件开发和知识管理等领域部署人工智能系统，了解这些系统的重要性日益凸显。根据输出性质对多模态系统进行分类。这种分类有助于更好地进行架构设计、模型选择，并与下游用例保持一致。

Multimodal GenAI systems are distinguished not only by their ability to process diverse types of input, such as text, images, audio, or structured data, but also by the variety of outputs they are capable of producing. As organizations deploy AI systems across sectors like e-commerce, healthcare, software development, and knowledge management, it becomes important to classify multimodal systems based on the nature of their output. This classification allows for better architectural design, model selection, and alignment with downstream use cases.

本节介绍了一种基于多模态系统生成的输出类型对其进行分类的框架，重点关注六个核心类别：

This section introduces a framework for classifying multimodal systems based on the type of output they generate, focusing on six core categories:

• 文本转图像
• Text-to-image
• 图像转文本
• Image-to-text
• 文本和图像到图像
• Text and image-to-image
• 文本到规格和图像
• Text-to-specifications and image
• 文本转 SQL
• Text-to-SQL
• 文本转代码
• Text-to-code